You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "aminghadersohi (via GitHub)" <gi...@apache.org> on 2026/10/13 02:12:12 UTC
[PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
aminghadersohi opened a new pull request, #45232:
URL: https://github.com/apache/superset/pull/45232
### SUMMARY
Stopping a running Impala query in SQL Lab could fail the query instead of stopping it.
Once the operation's cancel handle was published, `sql_lab.cancel_query` cancelled the query through the coordinator's HTTP `/cancel_query` endpoint. The SQL Lab worker was still polling that same operation in `ImpalaEngineSpec.handle_cursor`, which https://github.com/apache/superset/pull/44737 extended to cancel the live cursor when the query is STOPPED. The HTTP cancel invalidates the handle under the worker, so `status()` raises, `handle_cursor` returns, and `fetch_data` fails with "Invalid or unknown query handle" before the Stop request has committed STOPPED. On Impala 4.5.2 this shows as:
- Stop on a running query: Stop returns 200, but the Run request returns HTTP 500 with the handle error.
- Stop on an admission-queued query whose handle is already published: Stop returns 422 and the query ends `failed`.
Fix: `ImpalaEngineSpec.prepare_cancel_query` always records the early-cancel flag, not only when no handle is published yet. The stop succeeds without the HTTP call, and `handle_cursor` cancels and closes the operation with `cancel_operation()` on its next poll. This works for pending/initialized operations as well as running ones.
Why not `has_implicit_cancel() -> True`? That flag is also read outside SQL Lab:
- The MCP worker would switch to `cursor.cancel()`, which impyla's `HiveServer2Cursor` does not have, so MCP deadline cancellation would silently stop working.
- `SQLExecutor._cancel_query` would return True without persisting anything the async worker can see, so an async executor stop would no longer reach Impala.
Routing through `prepare_cancel_query` changes only the two SQL Lab stop paths (`sql_lab.cancel_query` and `SQLExecutor._cancel_query`), both of which run `handle_cursor`. The HTTP `cancel_query` stays in place for chart and MCP cancellation, which have no polling loop.
Trade-off: a SQL Lab stop no longer cancels on the coordinator when no worker is polling the cursor at all (for example, a worker that has died). That matches how Hive and Presto stop SQL Lab queries.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A (backend only).
### TESTING INSTRUCTIONS
```
pytest tests/unit_tests/db_engine_specs/test_impala.py tests/unit_tests/db_engine_specs/test_impala_pending.py tests/unit_tests/db_engine_specs/test_cancel_query_validation.py tests/unit_tests/sql_lab_test.py tests/unit_tests/sql/execution
```
- `test_stop_running_query_uses_live_cursor` (SQL Lab and executor paths): with a published cancel handle, Stop makes no HTTP call, and `handle_cursor` on a RUNNING operation cancels and closes it through the cursor. It fails on master, where the HTTP cancel is sent.
- `test_prepare_cancel_query_flags_stop`: the flag is recorded with or without a published handle.
- Existing pending-state tests (`test_stop_before_cancel_id_is_published_uses_live_cursor`, `test_cancel_unfinished_operation`, `test_stopped_status_cancels_pending_operation`) still pass, so Stop on an admission-queued query is still cancelled through the live cursor.
Manual: on an Impala-backed database, run a long query in SQL Lab (for example a CTAS with `sleep()`), press Stop while it is running, and confirm the query ends as stopped with no error on the Run request. Repeat while the query is waiting in admission control.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6145346748
Re-review of the full diff against master at `04cb0dad0f`, covering the Impala spec, the Celery finalization and error-recovery stop races, Impala/Hive cursor behaviour, and test coverage.
**Findings**
- **MEDIUM (fixed in `ab5df9b3bc`):** `ImpalaEngineSpec.prepare_cancel_query` published `STOPPED` unconditionally. Cancelling an async Impala query after it had finished (`SQLExecutor` handle `cancel()`, or a SQL Lab stop that lost the race with the worker's terminal commit) relabelled a `SUCCESS`, `FAILED` or `TIMED_OUT` query as `STOPPED`. `get_result()` then stopped returning the results it had already stored. The stop is now claimed with a conditional `UPDATE ... WHERE status NOT IN (success, failed, timed_out)`. That write holds the row lock until commit, so it serialises with the worker's guarded terminal commit (`_commit_query_unless_stopped`) in either order. When the query has already finished, the stop flag is not set and the stored status is left alone.
- Regression test: `test_prepare_cancel_query_keeps_finished_status`, which uses a real SQLite session and a stale `RUNNING` read on the caller side. Its finished-status cases fail without the fix.
No other valid findings:
- The 04cb0dad fix for error-recovery autoflush works. The task runs under `no_autoflush`, and the only paths that write terminal metadata are the guarded commit and `_stopped_query_payload`.
- Under REPEATABLE READ, the stop checks that preserve pending metadata can read a stale snapshot. The guarded conditional `UPDATE` does a current read, so correctness never depends on those checks.
- When a stop arrives while results are being written, a results-backend entry can be left without a reference. It expires with the cache timeout, so it is harmless.
**Tests**
- Ran the Impala and Hive engine-spec tests, `tests/unit_tests/sql/execution/`, SQL Lab, DAO, queries and the stacktrace tests: 588 passed. The impyla-backed stop tests were included.
- The only failures were 8 `generate_series` executor cases, which need `duckdb`. That package isn't installed in this environment, and those cases fail the same way without this change.
- pre-commit passes on the changed files, checked against the pinned ruff 0.9.7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6151148844
AI review at ab5df9b3 (full PR diff vs master), fixed in 386b5e0b.
**MEDIUM: a stop that loses the race with completion still relabels the finished query as STOPPED.** When `ImpalaEngineSpec.prepare_cancel_query` declines to claim the stop (the worker has already committed SUCCESS/FAILED/TIMED_OUT), it refreshes the query and returns. But `sql_lab.cancel_query` then carried on to the coordinator HTTP cancel for the finished operation (`QUERY_CANCEL_KEY` is set by then). If that returned True, `QueryDAO.stop_query` wrote STOPPED and a new `end_time` over the terminal row. That is the same overwrite ab5df9b3 fixed inside `prepare_cancel_query`, now one layer up, and it also reintroduced the HTTP cancel this PR removes. `SQLExecutor._cancel_query` had the same fall-through, though it does not write a status, so there it only sent a spurious HTTP cancel.
Fix (386b5e0b): after `prepare_cancel_query`, `cancel_query` returns without cancelling if the query is finished, `stop_query` re-checks and keeps the terminal status, and `SQLExecutor._cancel_query` returns False. Regression tests: `tests/unit_tests/dao/queries_test.py::test_query_dao_stop_query_keeps_status_when_impala_finishes_first` (runs the real Impala `prepare_cancel_query` against sqlite, with the worker's terminal commit landing between the DAO read and the claim) and `tests/unit_tests/sql/execution/test_executor.py::test_cancel_query_finished_while_preparing`. Both fail on ab5df9b3 and pass on 386b5e0b.
Also checked, no findings:
- Celery finalization: the guarded conditional UPDATE in `_commit_query_unless_stopped` against the Impala claim UPDATE serialises on the row lock in both orders.
- `no_autoflush` around error recovery, and the `preserve_pending` refresh, which discards pending success metadata once a stop is observed.
- `_store_results_in_backend` after a stop: the orphaned results key only expires via TTL.
- Other engines: Trino's `prepare_cancel_query` is unchanged. Engines without a prepare hook never refresh status, so the new re-check is a no-op for them.
- Legacy `sql_lab.execute_sql_statements` already preserves STOPPED.
Tests run: `tests/unit_tests/dao/`, `tests/unit_tests/sql/execution/`, `test_impala.py`, `test_impala_pending.py`, `test_sqllab_error_stacktrace.py`, `sql_lab_test.py`. 605 passed. The only 8 failures are `generate_series` cases that need `duckdb`, which is not installed locally; they fail the same way without this change. pre-commit is clean on the changed files. Trino unit tests were not run locally (`trino` driver not installed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6161699002
Product acceptance at `ce10e4f068174d2009d7beefa740c2cdc339d12c` (regressions checked against merge-base `1cedab01e9e2513e1c7548590a451c6434987b52`).
**Verdict: ship.** No defects found in the diff; no commits pushed.
Environment: real Impala 4.5.2 (one impalad, one admission slot, HS2 binary and HTTP transport). The metadata DB was MySQL 8.0.46 with `REPEATABLE READ` (verified via `@@transaction_isolation`), and a Postgres-metadata subset was also run. Celery prefork and gunicorn ran this branch. I drove the SQL Lab REST API, and real SQL Lab in Chromium. The Impala web UI and instrumented `cancel_operation` / `ImpalaEngineSpec.cancel_query` counted every cancel.
| Case | This PR | master |
|---|---|---|
| Stop a running query (sync, async, HTTP transport, browser) | 15/15 + 3/3 browser `stopped`; Impala cancelled in ~1.8s; 1 cursor cancel, 0 HTTP | 6/6 `failed`, sync Run 500, 1 HTTP cancel |
| Stop an admission-queued query | 26/26 + 2/2 browser `stopped`, Run returns in ~1.4s | 6/6 `failed` |
| Two concurrent Stops / Stop pressed twice | 6/6 and 4/4: both 200, still one cursor cancel | – |
| Stop after completion, plus a 60-run race sweep near completion | 8/8 keep `success`; 60/60 Run response matches the final row; stop on a `failed` query keeps `failed` 4/4 | – |
| `execute_async` + `handle.cancel()` (running / queued) | 3/3 + 3/3 `stopped`, cancelled through the cursor | running ran 30s to `success`; queued `failed` with unknown query handle |
| Chart-style HTTP cancel (`cancel_query` with the chart-task stub) | 3/3: one HTTP cancel, `prepare_cancel_query` not called | – |
| Postgres / MySQL datasources | Postgres 18/18; MySQL complete/error 5/5 | same |
| Postgres metadata DB subset | 24/24 | – |
Things I found that are **pre-existing** (they reproduce identically at the merge-base and are not caused by this PR):
- **Timeouts on Impala.** A sync `SQLLAB_TIMEOUT` raised while `handle_cursor` is polling is swallowed by its broad `except Exception`, so the Run waits for the whole query and then fails from `fetch_data`'s deadline. With an async soft time limit, the cursor is cancelled but the task hangs until the hard limit and the row stays `running`. Separately, `get_sql_results` hardcodes `soft_time_limit=21600`, so `SQLLAB_ASYNC_TIME_LIMIT_SEC` is not applied.
- **A Stop during Impala's fetch phase is not sent to Impala.** On a streaming select, Impala reports `FINISHED` (rows available) almost at once and does the work during fetch. With this PR the UI and the row say `stopped` right away, but Impala finishes the query (15s) and a sync Run blocks until then. On master, Stop returns 422 and the query ends `failed`, and the Run still blocks ~15s. So this is not worse, and the user-facing result is better. A possible follow-up is a fetch loop that checks for the stop.
- A stopped Impala CTAS leaves its target table behind (24/24 here, 6/6 master).
- Stop on a MySQL datasource ends `failed` with "Lost connection to MySQL server during query" (6/6 here, 4/4 master).
- `SQLExecutor` handle `get_result()` cannot decode stored results ("'utf-8' codec can't decode byte 0x85") for Impala and Postgres (10/10 here, 3/3 master).
Not verified:
- Ctrl+E and window-close (`cancel_query_on_windows_unload`): headless Chromium sent no stop request, and the frontend is unchanged by this PR.
- MCP deadline cancellation end-to-end.
- Hive, Presto and Trino.
- The frontend assets came from the `apache/superset:master` image built 2026-10-09.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6160026844
CI diagnosis at `ce10e4f068174d2009d7beefa740c2cdc339d12c`:
- **Docker verification: infrastructure failure.** https://github.com/apache/superset/actions/runs/38496035494 failed while `supersetbot` fetched repository tags: GitHub returned HTTP 500 for `GET /repositories/39464018/tags?per_page=100&page=6`. All three image builds (lean, superset, dev) succeeded. Nearby master run https://github.com/apache/superset/actions/runs/38499336505 and another PR run https://github.com/apache/superset/actions/runs/38496214790 succeeded.
- **Unit job: treating as likely runner/infra timeout, not a reproduced branch regression.** https://github.com/apache/superset/actions/runs/38496035611/job/115632873116 concluded `cancelled`; its annotation says it exceeded the 45-minute execution limit. The required check failed because its dependency was cancelled. The unit log blob is unavailable (HTTP 404 `BlobNotFound`), and no JUnit artifact was uploaded, so the exact stalled point cannot be established. Nearby master https://github.com/apache/superset/actions/runs/38499336658 passed 27,035 tests in 28m29s; https://github.com/apache/superset/actions/runs/38496214881 passed the same count in 23m55s. Both passed the scoped coverage gates. The previous branch head also passed in https://github.com/apache/superset/actions/runs/38483021880.
- Local verification at the unchanged head: **470 passed in 17.45s**, covering Impala (including impyla-backed stop cases), cancellation validation, SQL Lab, query DAO, and SQL execution/Celery. No hang reproduced. Pre-commit, including mypy, passed on all ten branch-changed files with the repository-pinned Ruff 0.9.7. Used the existing acceptance virtualenv with both impyla and duckdb installed.
Reran failed jobs **once** in each affected workflow above; both are attempt 2. No code change or push; draft state and reviews are unchanged.
No completed §16 acceptance verdict was found in PR comments or the `agent-smith` KB. The previous acceptance session ended mid-testing (its last update concerned a REPEATABLE READ snapshot in the test harness), without a final verdict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6144316203
Reviewed the full diff against apache/superset master at `447eb1e57283d724e4fd9a68a3e72a7958e29ef3`, after verifying the live head matched the requested SHA.
- **MEDIUM — Celery exception recovery could bypass the terminal stop guard.** The real `_get_query` in `execute_sql_task` autoflushed dirty worker state before `_handle_query_error` checked STOPPED. A concurrent soft time limit could overwrite STOPPED with TIMED_OUT; a backend exception could overwrite the recorded stop time and cancellation metadata. ORM loads during OAuth retry evaluation exposed the same problem. Three regression cases using independently committed SQLAlchemy sessions failed before the fix. Closed by **04cb0dad0f62706416af659faa8bc6f87be2b88b**: suppress implicit autoflush across task execution and error recovery, retaining explicit guarded commits. Eight regression/control cases verify stopped results and normal FAILED/TIMED_OUT behavior with the real query lookup.
Rechecked Impala running/admission-queued live-cursor cancellation, synchronous stop/fetch handling, partial results, late serialization/backend stops, atomic SUCCESS/FAILED finalization, and the earlier fixes. Chart/MCP cancellation still makes a single HTTP request and does not use SQL Lab's preparation hook. No additional reproducible findings.
Validation:
- **651 focused tests passed**, including `pytest tests/unit_tests/sql_lab* tests/unit_tests/db_engine_specs/test_impala.py`, SQL execution/Celery, Impala pending/cancellation validation, Hive, Presto, chart cancellation, MCP workers, and stacktrace regressions.
- **2,338 SQL tests passed**, with **100% statement/branch coverage**.
- Chart/MCP Impala HTTP smoke probe passed with mocked RPCs.
- Pre-commit, including mypy, passed on the two changed files and all five branch-changed files, using repository-pinned Ruff.
- Real-Impala acceptance was not run.
Pushed normally to the designated fork branch. Verified live head `04cb0dad0f62706416af659faa8bc6f87be2b88b`; draft state and master base are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6157202346
AI code review of this PR at 386b5e0b (full diff against master), follow-up to the review at ab5df9b3.
**Finding (LOW–MEDIUM, fixed in 970341c265):** when `ImpalaEngineSpec.prepare_cancel_query`'s conditional `UPDATE ... SET status=STOPPED` matched no row because the worker had already committed SUCCESS, FAILED, or TIMED_OUT, the plain `db.session.refresh(query)` could still return the RUNNING snapshot from the stop request's first SELECT. This only happens when the metadata DB runs under REPEATABLE READ, which is not the default but is configurable through `SQLALCHEMY_ENGINE_OPTIONS`. In that case the post-prepare re-checks in `sql_lab.cancel_query`, `QueryDAO.stop_query`, and `SQLExecutor._cancel_query` did not see the terminal status. They fell through to the HTTP cancel, and `stop_query` then overwrote the finished query with STOPPED. The fix uses `refresh(query, with_for_update=True)`, the same locking read `_commit_query_unless_stopped` already uses. The regression test `test_finished_query_stop_bypasses_repeatable_read_snapshot` covers the SQL Lab and executor callers for
all three terminal statuses. All 6 cases fail on 386b5e0b and pass after the fix.
**Checked, no further findings:**
- Stop vs. worker terminal commit: both are conditional UPDATEs, so the row lock serialises them and the losing side re-evaluates against the committed row.
- Callers after prepare: `sql_lab.cancel_query` returns True for a finished query and `QueryDAO.stop_query` returns without writing STOPPED. `SQLExecutor._cancel_query` returns False. No other caller of `sql_lab.cancel_query` exists.
- Trino, Presto, and Hive: their cancel paths are unchanged by this PR. Trino's `prepare_cancel_query` keeps its flag-only behaviour. MCP and chart cancellation go through `tasks/query_cancel.py` and still use the Impala HTTP cancel, which matches the `has_implicit_cancel` docstring.
- Celery stop checks (`preserve_pending`, preserving TIMED_OUT in `_handle_query_error`, `no_autoflush` around finalization).
**Tests:** test_impala_pending, test_impala, dao/queries_test, sql/execution/test_celery_task, test_executor, test_sqllab_error_stacktrace, and sql_lab_test: 430 passed. The 8 `generate_series` executor failures come from `duckdb` not being installed locally and are unrelated to this PR. `pre-commit run` on the changed files passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4267805409
##########
superset/sql/execution/celery_task.py:
##########
@@ -481,25 +544,44 @@ def _execute_sql_statements(
)
) from ex
- # Check if stopped
- if not execution_results:
- payload.update({"status": QueryStatus.STOPPED.value})
- return payload
+ # Cancelling can clear cursor.description, leaving a nonempty result
+ # list that looks like successful DML. Check intent, not just results.
+ stopped = not execution_results or check_stopped()
# Commit for mutations
- if parsed_script.has_mutation() or query.select_as_cta:
+ if not stopped and (parsed_script.has_mutation() or query.select_as_cta):
conn.commit() # pylint: disable=consider-using-transaction
Review Comment:
Valid. If a Stop committed after `check_stopped()` but while `conn.commit()` ran, the mutation/CTAS was durable but the post-commit re-check returned the stopped payload and `_commit_query_unless_stopped` refused to write SUCCESS.
Fixed in fbbeb9f178015b29b915dce58299c7a92a981b8e: once `conn.commit()` succeeds, later stop checks are skipped (`_never_stopped`), and `_store_results_in_backend` / `_publish_query_result` take `durable=True` and write the terminal status unconditionally. A Stop that lands before the commit still prevents the commit, as before.
Regression test: `tests/unit_tests/sql/execution/test_celery_task.py::test_stop_during_mutation_commit_does_not_mask_committed_work` (real SQLite metadata DB; the Stop is committed from an independent session inside `conn.commit()`; parametrized for no results backend, backend write success, and backend write failure). It fails on ce10e4f068 (status `stopped`) and passes on fbbeb9f178.
Sibling paths checked:
- `superset/sql/execution/celery_task.py` `_execute_sql_statements`: fixed (the only target-DB commit in the new path).
- `superset/sql/execution/executor.py` `execute_sql_with_cursor`: checks stop before each statement and does not commit the target connection, not affected.
- `superset/sql_lab.py` legacy `execute_sql_statements` (`conn.commit()` then refresh/STOPPED check): same pattern, but that code is unchanged from master by this PR; not fixed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6173533153
@rebenitez1802 thanks for the thorough review. I split the PR and worked through each finding. Head is 70fec12f161c952b7fc52b7244fc62dc1063077f.
**🟡 Description no longer matches the diff.** I split it. 8825befaa584ab83c18aad509eefdad83931057a moves the all-engine Celery finalization rework and the `sql_lab.py` `durable`/`on_commit` changes out of this PR; they are now https://github.com/apache/superset/pull/45283. This PR keeps the Impala routing, the finished-status checks it needs, and the one Celery change Impala executor stops depend on: a cancelled impyla cursor clears `cursor.description`, so the stop has to be read from status or the early-cancel flag (`test_cancelled_impala_cursor_is_not_finalized_as_success`). I rewrote the description to cover every behaviour change left in the diff, including that Celery change and its two extra metadata reads.
**🟡 A stop no longer reaches Impala when the worker isn't polling.** Not fixed in code; the trade-off stands and needs your sign-off. Documented in 34f6466f98de680ee9a144d86f6c5df0467f1bf2 (the `prepare_cancel_query` docstring lists dead or restarted workers, a stop during `fetch_data`, and polling that ended because `cursor.status()` raised), in 8b8924e9aa (UPDATING.md note for operators who relied on the HTTP cancel), and in the description, which also names the removed `test_stop_with_cancel_id_uses_http` guarantee.
**🟢 `stop_query` writes STOPPED again after Impala's claim.** Fixed in 34f6466f98de680ee9a144d86f6c5df0467f1bf2. `stop_query` writes nothing once the engine spec has committed the stop, and otherwise writes STOPPED with a conditional UPDATE that skips finished rows. Impala's claim records the end time itself, and the stale comment is updated. One correction to the finding: the status itself was not rewritten, because SQLAlchemy leaves an unchanged column out of the UPDATE, but `end_time` was overwritten. Regression tests, both real SQLite sessions with a second session as the worker, failing before and passing after:
- `test_stop_query_does_not_rewrite_impala_claim`: the worker commits SUCCESS with an end time just before `stop_query` commits. Before the fix the end time was overwritten.
- `test_stop_query_keeps_status_committed_after_its_check`: the same race for any engine. Before the fix the row ended `stopped` instead of `success`.
**🟢 A declined claim leaves a `FOR UPDATE` lock open.** Fixed in 34f6466f98de680ee9a144d86f6c5df0467f1bf2: the declined branch commits after the locking read. Regression test: `test_declined_impala_claim_ends_transaction` (SQL Lab and executor paths). A second connection writes the row after the declined stop. Before the fix that write failed with `database is locked`.
**🟢 `_handle_query_error` raises after committing.** This moved with the split, so it is fixed in https://github.com/apache/superset/pull/45283 (95efdd91ee): the status enum is computed before the commit and `expire_on_commit=False` is dropped from the tests. With production-style sessions the 14 cases you found failed with `AttributeError` before the fix and pass after.
**🟢 `durable` relies on `conn.commit()`.** Moved to https://github.com/apache/superset/pull/45283. A comment there (d117709fde) notes that autocommit engines make DML and CTAS permanent at statement completion, so a stop before the pre-commit check still ends STOPPED. That PR's description also lists it as a known limitation. Not fixed.
**🟢 Redundant metadata reads.** Moved to https://github.com/apache/superset/pull/45283 (d117709fde). It drops the post-cleanup and `_finalize` re-checks. A successful SELECT goes from 11 to 9 statements in my harness (master: 5): one full refresh, two status-only reads, the guarded UPDATE and the final write. `test_successful_select_reads_stop_state_once_per_checkpoint` fails before (3 full refreshes) and passes after (1). The two status reads stay because the guarded UPDATE only checks status, and an executor stop can publish only the early-cancel flag. In this PR the Celery path keeps the two reads described above.
**🟢 Finished-status set duplicated; return values disagree.** Fixed in 34f6466f98de680ee9a144d86f6c5df0467f1bf2. `FINISHED_QUERY_STATUSES` lives in `superset/common/db_query_status.py` and is used by `impala.py`, `executor.py`, `sql_lab.py` and `daos/query.py`. `sql_lab.cancel_query` now returns False like `SQLExecutor._cancel_query` for a query that finished while the stop was prepared; `stop_query` checks the finished status before raising. Covered by `test_finished_query_stop_bypasses_repeatable_read_snapshot`, which asserts False on both paths.
**🟢 Tests that build the fix into their mocks.**
- `test_poll_sees_stop_committed_after_its_snapshot` is replaced by a real-session version in `tests/unit_tests/dao/query_stop_isolation_test.py`. It uses SQLite in WAL mode with explicit `BEGIN`, so the worker's transaction keeps its first snapshot. With the per-poll commit from ce10e4f068 reverted it fails (`cancel_operation` not called); at head it passes.
- `test_finished_query_stop_bypasses_repeatable_read_snapshot` stays mocked. SQLite cannot model InnoDB's current read on a stale snapshot; it fails the write with `SQLITE_BUSY_SNAPSHOT`. The docstring now says so. Remaining: nothing in unit tests protects the locking read on MySQL or Postgres; only the real-Impala + MySQL acceptance run does.
- `test_publish_query_result_preserves_failure` moved with the split and now uses a real session with the FAILED status committed first, as `_store_results_in_backend` does.
**🟢 Real-Impala acceptance predates the last commits.** Not done yet. The Impala 4.5.2 + MySQL acceptance matrix has not been re-run at 70fec12f161c952b7fc52b7244fc62dc1063077f. Remaining: a re-run at head.
**🟢 Red `check-openapi-spec-drift`.** Merged master in 1713bbfe06 (merge commit, no rebase). CI is running at the new head.
Siblings checked:
- superset/sql_lab.py:891 — fixed in 34f6466f98: uses `FINISHED_QUERY_STATUSES` and returns False.
- superset/sql/execution/executor.py:1465 — fixed in 34f6466f98: uses `FINISHED_QUERY_STATUSES`.
- superset/daos/query.py:73 — fixed in 34f6466f98: both finished checks use the shared constant and the STOPPED write is conditional.
- superset/db_engine_specs/trino.py:374 — not affected: checks STOPPED or TIMED_OUT, a different set.
- superset/commands/sql_lab/execute.py:137 — not affected: checks RUNNING, PENDING or TIMED_OUT, a different set.
- superset/sql/execution/executor.py:545 — not affected: builds a TIMED_OUT error result, not a status set.
Searched with `grep -rn "FINISHED_QUERY_STATUSES\|QueryStatus.TIMED_OUT,$\|refresh(.*with_for_update" superset`. The `with_for_update` refresh in https://github.com/apache/superset/pull/45283 (`_commit_query_unless_stopped`) is followed by `_stopped_query_payload`, which commits, so it does not hold the lock.
Tests: `pytest tests/unit_tests/db_engine_specs/test_impala.py tests/unit_tests/db_engine_specs/test_impala_pending.py tests/unit_tests/db_engine_specs/test_cancel_query_validation.py tests/unit_tests/sql_lab_test.py tests/unit_tests/sql/execution tests/unit_tests/dao tests/unit_tests/test_sqllab_error_stacktrace.py tests/unit_tests/queries`: 677 passed. 8 `test_apply_limit_generate_series_returned_rows` cases fail locally because `duckdb` is not installed here; they are not touched by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6201791722
In response to https://github.com/apache/superset/pull/45232#pullrequestreview-5537157969 :
**Async-cancel scope:** The scope expansion is fixed in daf2d551660983edb8a8f38f150fdb3619dbc4a4. I reverted the executor-owned STOPPED write, the Celery error-handler stop recheck, the developer-doc section and their persistence tests in daf2d551660983edb8a8f38f150fdb3619dbc4a4. The error handler is byte-identical to master. I retained `test_stop_seen_during_execution_skips_mutation_commit` (UPDATE/CTAS, both stop signals).
Regression: `test_executor_cancel_does_not_commit_caller_session` exercises PostgreSQL, MySQL, Hive and Presto through the real async handle and separate SQLite metadata sessions, including unrelated pending caller work. Together with the strengthened `test_cancel_query_early_cancel_flag`, all five cases fail against 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c and pass here. The affected suites pass 2,753 tests, with the Impala driver installed; pre-commit passes on branch-changed files.
Siblings checked:
- superset/sql/execution/executor.py:1275 — fixed in daf2d551660983edb8a8f38f150fdb3619dbc4a4
- superset/sql/execution/executor.py:1430 — fixed in daf2d551660983edb8a8f38f150fdb3619dbc4a4
- superset/sql/execution/executor.py:1469 — fixed in daf2d551660983edb8a8f38f150fdb3619dbc4a4
- superset/sql/execution/celery_task.py:76 — fixed in daf2d551660983edb8a8f38f150fdb3619dbc4a4
- superset/sql/execution/celery_task.py:371 — fixed in daf2d551660983edb8a8f38f150fdb3619dbc4a4
- superset/sql_lab.py:873 — not affected: legacy cancellation delegates metadata ownership to QueryDAO, not the async handle.
- superset/daos/query.py:79 — not affected: its existing stop claim and terminal-outcome tests remain unchanged.
- superset/db_engine_specs/impala.py:244 — not affected: its engine-specific claim already commits STOPPED; the live-cursor tests still pass.
- superset/db_engine_specs/trino.py:527 — not affected: its pre-existing prepare hook owns the early-flag commit; the generic executor adds no commit.
- superset/db_engine_specs/hive.py:665 — not affected: implicit-cancel capability is unchanged; the Hive regression case pins caller transaction ownership.
- superset/db_engine_specs/presto.py:1477 — not affected: implicit-cancel capability is unchanged; the Presto regression case pins caller transaction ownership.
- superset/mcp_service/worker.py:794 — not affected: MCP uses its own cancellation path, not SQLExecutor cancellation.
**Merge coupling:** the common stop-before-engine-cancel design belongs in https://github.com/apache/superset/pull/45283 . Merge order: that PR first, then this PR after merging `origin/master` into this branch; I am not duplicating that redesign here.
Remaining: this branch is not yet stacked on the shared design. `git merge-tree --write-tree HEAD origin/pr-45283` exits 1 with conflicts in `UPDATING.md`, `daos/query.py`, `impala.py`, `celery_task.py` and `test_celery_task.py`. General async/error/post-commit cancellation behavior belongs in that PR. Whether its HTTP-only Impala path avoids both reported failures without the polling trade-off still needs real-Impala acceptance; I cannot claim that result from unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "rebenitez1802 (via GitHub)" <gi...@apache.org>.
rebenitez1802 commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4274734056
##########
superset/daos/query.py:
##########
@@ -70,26 +70,50 @@ def stop_query(client_id: str) -> None:
if not query:
raise QueryNotFoundException(f"Query with client_id {client_id} not found")
- if query.status in [
- QueryStatus.FAILED,
- QueryStatus.SUCCESS,
- QueryStatus.TIMED_OUT,
- ]:
+ if query.status in FINISHED_QUERY_STATUSES:
logger.warning(
"Query with client_id could not be stopped: query already complete",
)
return
- if not sql_lab.cancel_query(query):
+ cancelled = sql_lab.cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ # Preparing the stop can observe the worker's terminal commit.
+ logger.warning(
+ "Query with client_id could not be stopped: query already complete",
+ )
+ return
+ if not cancelled:
raise SupersetCancelQueryException("Could not cancel query")
+ if query.status == QueryStatus.STOPPED:
+ # The engine spec already committed the stop while preparing it
+ # (see ImpalaEngineSpec.prepare_cancel_query).
+ return
+
# cancel_query() may have staged an early-cancel flag on query.extra
# without committing it (see its docstring/comments); the
# @transaction decorator commits it together with status=STOPPED
# below in one transaction, closing the window where another
# request could observe the flag set but the status still RUNNING.
- query.status = QueryStatus.STOPPED
- query.end_time = now_as_float()
+ #
+ # The write is conditional: the worker can commit a terminal status
+ # after the check above, and that result must not be overwritten.
+ stopped = (
+ db.session.query(Query)
+ .filter(
+ Query.id == query.id,
+ Query.status.notin_(FINISHED_QUERY_STATUSES),
Review Comment:
**🔴 High — Conditional STOPPED write turns a stop into FAILED on non-Impala engines**
This skips the STOPPED write when the row is in `FINISHED_QUERY_STATUSES`, and that set includes FAILED. Engines whose cancel kills the statement from outside the worker are affected: Postgres (`pg_terminate_backend`), MySQL (`KILL CONNECTION`), Snowflake, Redshift, Oracle, and Trino once the query id is published.
- The worker's statement fails straight away, so its error handler can commit FAILED before this UPDATE runs.
- The UPDATE then matches no row. The Stop request returns 200, but the query ends `failed` with the driver's "terminated" error in Query History and the API.
- Master's unconditional write ended STOPPED whichever side committed first.
We reproduced it with the `test_stop_query_keeps_status_committed_after_its_check` harness by having the worker commit FAILED inside the cancel call: the query ended `failed` at head and `stopped` with master's `daos/query.py`. Impala never reaches this write; it returns early after its own claim at `:89`.
This came from my round-1 suggestion to make the write conditional; I should have said only statuses a stop can't cause should be protected. Suggested fix below. Please also add a FAILED case to `query_stop_isolation_test.py`, where the worker commits FAILED during `cancel_query`. Alternatively, move this conditional write to #45283 and keep only the Impala early return here.
```suggestion
# The write is conditional: the worker can commit SUCCESS or TIMED_OUT
# after the check above, and that result must not be overwritten. A
# FAILED committed in this window is usually caused by the cancel
# itself, so it is still replaced with STOPPED.
stopped = (
db.session.query(Query)
.filter(
Query.id == query.id,
Query.status.notin_((QueryStatus.SUCCESS, QueryStatus.TIMED_OUT)),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4276297271
##########
superset/daos/query.py:
##########
@@ -70,26 +70,50 @@ def stop_query(client_id: str) -> None:
if not query:
raise QueryNotFoundException(f"Query with client_id {client_id} not found")
- if query.status in [
- QueryStatus.FAILED,
- QueryStatus.SUCCESS,
- QueryStatus.TIMED_OUT,
- ]:
+ if query.status in FINISHED_QUERY_STATUSES:
logger.warning(
"Query with client_id could not be stopped: query already complete",
)
return
- if not sql_lab.cancel_query(query):
+ cancelled = sql_lab.cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ # Preparing the stop can observe the worker's terminal commit.
+ logger.warning(
+ "Query with client_id could not be stopped: query already complete",
+ )
+ return
+ if not cancelled:
raise SupersetCancelQueryException("Could not cancel query")
+ if query.status == QueryStatus.STOPPED:
+ # The engine spec already committed the stop while preparing it
+ # (see ImpalaEngineSpec.prepare_cancel_query).
+ return
+
# cancel_query() may have staged an early-cancel flag on query.extra
# without committing it (see its docstring/comments); the
# @transaction decorator commits it together with status=STOPPED
# below in one transaction, closing the window where another
# request could observe the flag set but the status still RUNNING.
- query.status = QueryStatus.STOPPED
- query.end_time = now_as_float()
+ #
+ # The write is conditional: the worker can commit a terminal status
+ # after the check above, and that result must not be overwritten.
+ stopped = (
+ db.session.query(Query)
+ .filter(
+ Query.id == query.id,
+ Query.status.notin_(FINISHED_QUERY_STATUSES),
Review Comment:
Correction to my previous reply: one Trino path was still not fixed. A Trino stop issued before the query ID is published commits only the early-cancel flag. The worker could see the flag, kill the statement and commit FAILED before `stop_query` read the row back, so Stop returned success and the query stayed FAILED.
Fixed in 4a2cc30492ddd960bca72388122c119fd9a914eb: Trino's `handle_cursor` records STOPPED once its kill succeeds, before the killed statement raises, so `handle_query_error` keeps it. A failed kill records nothing, and TIMED_OUT is not changed.
Regression tests (fail at 8d0fcf8ab3, pass at 4a2cc30492):
- `tests/unit_tests/dao/query_stop_isolation_test.py::test_trino_early_cancel_that_kills_the_worker_ends_stopped` (real DAO, real Trino `execute_with_cursor`/`handle_cursor`, real `handle_query_error`, worker in a second SQLite session; ended `failed` before)
- `tests/unit_tests/db_engine_specs/test_trino.py::test_handle_cursor_early_cancel_records_stop_only_when_killed[running-True-stopped]`
Controls that pass before and after: `test_trino_early_cancel_keeps_failure_that_predates_it` (a FAILED committed before the flag stays FAILED with its error), and the `[running-False-running]` (kill failed, no STOPPED) and `[timed_out-True-timed_out]` cases.
Siblings checked:
- superset/db_engine_specs/trino.py:406 — fixed in 4a2cc30492ddd960bca72388122c119fd9a914eb
- superset/db_engine_specs/presto.py:1431 — not affected: implicit cancel; the worker cancels only after it reads a committed STOPPED, and `handle_query_error` keeps STOPPED
- superset/db_engine_specs/hive.py:448 — not affected: same implicit cancel on a committed STOPPED; covers Spark and DatabricksHive, which subclass Hive
- superset/db_engine_specs/impala.py:244 — not affected: `prepare_cancel_query` commits STOPPED with the flag before the worker can act; covers Ascend, which subclasses Impala
- superset/db_engine_specs/databricks.py:255 — not affected: no cancel overrides, so it takes the generic no-ID path below
- superset/sql_lab.py:937 — not affected: the generic no-ID path stages the flag uncommitted and `stop_query` commits it together with STOPPED, so the worker never sees the flag alone
- superset/daos/query.py:109 — not affected: Postgres, MySQL, Snowflake, Redshift, Oracle, SingleStore, Ocient and Trino with a published ID kill from outside the worker, and the conditional write from the earlier commit replaces that FAILED (`test_stop_query_replaces_failure_caused_by_its_cancel`)
Remaining: the async executor chain (`AsyncQueryHandle.cancel` -> `SQLExecutor._cancel_async_query` -> `_cancel_query`) still returns success without writing STOPPED, and its worker error handler `celery_task._handle_query_error` writes FAILED over STOPPED. So an external kill, or a Trino early-flag kill, through that executor still ends FAILED. This is the same on master and is not fixed in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi merged PR #45232:
URL: https://github.com/apache/superset/pull/45232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4278022823
##########
superset/sql/execution/executor.py:
##########
@@ -1450,33 +1451,60 @@ def _cancel_query(database: Database, query: Any) -> bool:
:param query: Query model instance to cancel
:returns: True if cancelled successfully, False otherwise
"""
+ from superset.common.db_query_status import FINISHED_QUERY_STATUSES, QueryStatus
from superset.constants import QUERY_CANCEL_KEY, QUERY_EARLY_CANCEL_KEY
+ from superset.models.sql_lab import Query as QueryModel
from superset.utils.core import QuerySource
+ from superset.utils.dates import now_as_float
- # Some engines implicitly handle cancellation
- if database.db_engine_spec.has_implicit_cancel():
- return True
-
- # Some databases may need to make preparations for query cancellation
- database.db_engine_spec.prepare_cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ return False
- # Check early cancellation flag
- if query.extra.get(QUERY_EARLY_CANCEL_KEY):
- return True
+ cancelled = database.db_engine_spec.has_implicit_cancel()
+ if not cancelled:
+ database.db_engine_spec.prepare_cancel_query(query)
+ # The query may have finished while the stop was being prepared.
+ if query.status in FINISHED_QUERY_STATUSES:
+ return False
+ cancelled = bool(query.extra.get(QUERY_EARLY_CANCEL_KEY))
+
+ if not cancelled:
+ cancel_query_id = query.extra.get(QUERY_CANCEL_KEY)
+ if cancel_query_id is None:
+ return False
+
+ with database.get_sqla_engine(
+ catalog=query.catalog,
+ schema=query.schema,
+ source=QuerySource.SQL_LAB,
+ ) as engine:
+ with contextlib.closing(engine.raw_connection()) as conn:
+ with contextlib.closing(conn.cursor()) as cursor:
+ cancelled = database.db_engine_spec.cancel_query(
+ cursor, query, cancel_query_id
+ )
- # Get cancel ID
- cancel_query_id = query.extra.get(QUERY_CANCEL_KEY)
- if cancel_query_id is None:
+ if not cancelled:
return False
+ if query.status == QueryStatus.STOPPED:
+ # Impala commits its stop while preparing cursor-side cancellation.
+ return True
- # Execute cancellation
- with database.get_sqla_engine(
- catalog=query.catalog,
- schema=query.schema,
- source=QuerySource.SQL_LAB,
- ) as engine:
- with contextlib.closing(engine.raw_connection()) as conn:
- with contextlib.closing(conn.cursor()) as cursor:
- return database.db_engine_spec.cancel_query(
- cursor, query, cancel_query_id
- )
+ # Match QueryDAO.stop_query: preserve SUCCESS/TIMED_OUT committed while
+ # cancellation was in flight, but replace a FAILED caused by the kill.
+ stopped = (
+ db.session.query(QueryModel)
+ .filter(
+ QueryModel.id == query.id,
+ QueryModel.status.notin_((QueryStatus.SUCCESS, QueryStatus.TIMED_OUT)),
Review Comment:
Thanks @sadpandajoe, you're right: a stop committed before the killed statement raised was rewritten as FAILED by the Celery worker.
Fixed in 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c. `_handle_query_error` in `superset/sql/execution/celery_task.py` now does a status-only locking refresh (no autoflush, so the worker's pending changes aren't written first). If the row is STOPPED it rolls back and returns STOPPED. A local TIMED_OUT skips the check, so timeouts behave as before. The payload status is taken from the enum set in the handler, because after the commit the attribute reloads from a real session as a `str` and `.value` raised.
Regression test: `tests/unit_tests/sql/test_async_cancel.py::test_worker_error_after_committed_stop_keeps_stopped`. It runs `execute_sql_task` against a real SQLite metadata DB. The statement raises after STOPPED is committed, either by `SQLExecutor` `cancel()` in the worker's session or by a separate session (the web process). It asserts the committed row and the payload stay STOPPED, while a timeout stays TIMED_OUT and an error with no stop stays FAILED. At 8da595275cff9fe901f3e94ce2cfe7e6cf375ad2 with only the enum-payload change applied, the two stop cases fail with `assert 'failed' == 'stopped'`. With the guard, all 6 pass.
Siblings checked:
- superset/sql/execution/celery_task.py:82 — fixed in 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c
- superset/sql_lab.py:99 — not affected: `handle_query_error` (sync SQL Lab) already refreshes the status and keeps STOPPED.
- superset/db_engine_specs/trino.py:406 — fixed in 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c: the `_cancel_requested_query` docstring claim (error handler keeps the stop) now holds on the Celery path too.
- docs/developer_docs/extensions/development.md:254 — fixed in 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c: the cancellation docs now hold whether the worker or the cancel commits first.
- superset/sql/execution/celery_task.py:299 — not affected: the results-backend write failure runs after the stop re-check. A stop landing during finalization is the race handled in https://github.com/apache/superset/pull/45283.
- superset/sqllab/sql_json_executer.py:202 — not affected: this dispatch failure happens before any worker runs.
Remaining: https://github.com/apache/superset/pull/45283 replaces this function with a broader version, adding the early-cancel flag, discarding stored results and making a stop win over TIMED_OUT. Whichever merges second takes 45283's `_handle_query_error` and drops `_has_committed_stop`. The two already conflict in `celery_task.py` at 8da59527 (`_make_check_stopped_fn` and the stop finalization), so that resolution is needed either way.
##########
superset/sql/execution/celery_task.py:
##########
@@ -481,15 +484,23 @@ def _execute_sql_statements(
)
) from ex
- # Check if stopped
- if not execution_results:
- payload.update({"status": QueryStatus.STOPPED.value})
- return payload
+ # Cancelling can clear cursor.description, leaving a nonempty result
+ # list that looks like successful DML. Check intent, not just results.
+ stopped = not execution_results or check_stopped()
# Commit for mutations
- if parsed_script.has_mutation() or query.select_as_cta:
+ if not stopped and (parsed_script.has_mutation() or query.select_as_cta):
Review Comment:
Thanks @sadpandajoe, good catch: no test exercised that guard.
Fixed in 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c. Added `tests/unit_tests/sql/execution/test_celery_task.py::test_stop_seen_during_execution_skips_mutation_commit`, parametrized over an `UPDATE` script and a CTAS (`select_as_cta=True`). Stop intent appears during execution through either a STOPPED status or the early-cancel flag. Execution still returns a rowcount result, which is the case where the result list alone looks like successful DML. Each case asserts `conn.commit.assert_not_called()`, a STOPPED payload and status, and that success finalization never runs.
Regression check: with the guard removed locally (`if parsed_script.has_mutation() or query.select_as_cta:`), all 4 cases fail with `Expected 'commit' to not have been called. Called 1 times.` With the guard restored, all 4 pass.
Siblings checked:
- superset/sql/execution/celery_task.py:514 — fixed in 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c: the `not stopped` mutation/CTAS commit guard is now tested.
- superset/sql/execution/celery_task.py:518 — not affected: the post-connection `if stopped or check_stopped():` re-check is already covered by `test_stop_with_partial_results_skips_success_finalization`.
- superset/sql_lab.py:632 — not affected: this PR doesn't change it. In the sync path, a stop seen during a block raises `SqlLabQueryStoppedException` and returns before this `conn.commit()`.
Remaining: in the sync path, a stop that lands after the last block finishes is not re-checked before its `conn.commit()`. This PR doesn't touch that path, and I haven't changed it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4267230743
##########
superset/sql/execution/celery_task.py:
##########
@@ -481,25 +544,44 @@ def _execute_sql_statements(
)
) from ex
- # Check if stopped
- if not execution_results:
- payload.update({"status": QueryStatus.STOPPED.value})
- return payload
+ # Cancelling can clear cursor.description, leaving a nonempty result
+ # list that looks like successful DML. Check intent, not just results.
+ stopped = not execution_results or check_stopped()
# Commit for mutations
- if parsed_script.has_mutation() or query.select_as_cta:
+ if not stopped and (parsed_script.has_mutation() or query.select_as_cta):
conn.commit() # pylint: disable=consider-using-transaction
Review Comment:
**Suggestion:** `check_stopped()` runs before this separate database commit, so a concurrent Stop can be recorded while a mutation or CTAS still commits and is reported stopped.
**Assessment:** 🟠 `Major` · 🔁 `Occurrence: Rarely` · 🏷️ `Race condition`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=39fabd7613a64fea8d0e5a301afad879&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=39fabd7613a64fea8d0e5a301afad879&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/sql/execution/celery_task.py
**Line:** 552:553
**Comment:**
*Race Condition: `check_stopped()` runs before this separate database commit, so a concurrent Stop can be recorded while a mutation or CTAS still commits and is reported stopped.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45232&comment_hash=621cf4ebca62174b5696e68a7d6db142d90a94e53a2aa7dfab1e635edb7a86cf&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45232&comment_hash=621cf4ebca62174b5696e68a7d6db142d90a94e53a2aa7dfab1e635edb7a86cf&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4276476071
##########
superset/daos/query.py:
##########
@@ -70,26 +70,50 @@ def stop_query(client_id: str) -> None:
if not query:
raise QueryNotFoundException(f"Query with client_id {client_id} not found")
- if query.status in [
- QueryStatus.FAILED,
- QueryStatus.SUCCESS,
- QueryStatus.TIMED_OUT,
- ]:
+ if query.status in FINISHED_QUERY_STATUSES:
logger.warning(
"Query with client_id could not be stopped: query already complete",
)
return
- if not sql_lab.cancel_query(query):
+ cancelled = sql_lab.cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ # Preparing the stop can observe the worker's terminal commit.
+ logger.warning(
+ "Query with client_id could not be stopped: query already complete",
+ )
+ return
+ if not cancelled:
raise SupersetCancelQueryException("Could not cancel query")
+ if query.status == QueryStatus.STOPPED:
+ # The engine spec already committed the stop while preparing it
+ # (see ImpalaEngineSpec.prepare_cancel_query).
+ return
+
# cancel_query() may have staged an early-cancel flag on query.extra
# without committing it (see its docstring/comments); the
# @transaction decorator commits it together with status=STOPPED
# below in one transaction, closing the window where another
# request could observe the flag set but the status still RUNNING.
- query.status = QueryStatus.STOPPED
- query.end_time = now_as_float()
+ #
+ # The write is conditional: the worker can commit a terminal status
+ # after the check above, and that result must not be overwritten.
+ stopped = (
+ db.session.query(Query)
+ .filter(
+ Query.id == query.id,
+ Query.status.notin_(FINISHED_QUERY_STATUSES),
Review Comment:
I fixed the async executor cancel chain in ccb2d13b5987c2e581c6e7a419af85bffe88075e: successful cancellation commits STOPPED, replaces a FAILED caused by the kill, preserves SUCCESS/TIMED_OUT, and never writes STOPPED for a failed cancel.
Regression tests: `test_async_external_cancel_commits_only_successful_stop` uses real Postgres/MySQL cancel methods and a second metadata session; `test_async_cancel_commits_implicit_or_early_stop`, `test_async_cancel_exception_does_not_commit_stop`, and `test_async_cancel_leaves_finished_query_untouched` cover the other branches. `PYTHONPATH=.:superset-core/src /tmp/w45232-venv/bin/python -m pytest tests/unit_tests/dao/query_stop_isolation_test.py -k test_async_ -q` at 4a2cc30492ddd960bca72388122c119fd9a914eb: 10 failed, 12 passed; the complete module after: 32 passed.
Validation: the same pytest prefix with `tests/unit_tests/dao/query_stop_isolation_test.py tests/unit_tests/sql/execution/test_executor.py tests/unit_tests/sql/execution/test_celery_task.py -q`: 355 passed, 3 skipped; staged-file `PATH=/tmp/w45232-venv/bin:$PATH pre-commit run`: passed, including mypy.
Siblings checked: superset/sql/execution/executor.py — fixed in ccb2d13b5987c2e581c6e7a419af85bffe88075e; superset/sql_lab.py — not affected: QueryDAO persists successful sync stops; superset/daos/query.py — not affected: conditional write already preserves SUCCESS/TIMED_OUT; superset/queries/api.py — not affected: delegates to QueryDAO.stop_query; superset/tasks/query_cancel.py — not affected: chart/MCP cancellation stub has no SQL Lab Query row; superset/mcp_service/worker.py — not affected: delegates to cancel_chart_query; superset/tasks/async_queries.py — not affected: separate chart task lifecycle; superset/commands/tasks/reap.py — not affected: separate orphan task lifecycle. `rg` found no other production callers of SQLExecutor._cancel_query.
Remaining: the celery_task._handle_query_error overwrite of committed STOPPED, including async Trino early kills, belongs exclusively to https://github.com/apache/superset/pull/45283
At that PR's fetched fork/sql-stop-race-finalization head ba1fcc6b893a15a33f36c33fec060274b41e840c, `git show ...:superset/sql/execution/celery_task.py` shows _handle_query_error preserving committed STOPPED when durable=False, with a conditional commit guard for a concurrent stop; durable=True intentionally retains FAILED after warehouse commit. I did not change that handler here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6205849087
@rebenitez1802 thanks. What closed each heads-up, at c78bdb1577234511e7769fe099e4b6326cc768a9 (master merged, including https://github.com/apache/superset/pull/45283):
- `impala.py` conflict: Impala SQL Lab stops stay cursor-only. There is no HTTP cancel and no `QUERY_CANCEL_KEY` lookup on that path, so the `NameError` can't happen. `tests/unit_tests/db_engine_specs/test_impala_pending.py::test_stop_running_query_uses_live_cursor` and `tests/unit_tests/daos/test_query_dao.py::test_engine_early_cancel_commits_with_guarded_stop` pass at this head and fail with master's Impala preparation restored.
- `test_executor_cancel_does_not_commit_caller_session`: I kept master's committing `_cancel_async_query` and removed this test, since it asserted the opposite.
- `trino.py:413-415` nit: no longer applies. Master's Celery `_handle_query_error` now keeps accepted STOPPED outcomes too.
Siblings checked: `superset/sql_lab.py` and `superset/sql/execution/executor.py` — fixed in c78bdb1577234511e7769fe099e4b6326cc768a9; `superset/tasks/query_cancel.py` — not affected: chart and MCP cancellation call the engine HTTP cancel directly and bypass SQL Lab preparation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4268008357
##########
superset/sql/execution/celery_task.py:
##########
@@ -481,25 +544,44 @@ def _execute_sql_statements(
)
) from ex
- # Check if stopped
- if not execution_results:
- payload.update({"status": QueryStatus.STOPPED.value})
- return payload
+ # Cancelling can clear cursor.description, leaving a nonempty result
+ # list that looks like successful DML. Check intent, not just results.
+ stopped = not execution_results or check_stopped()
# Commit for mutations
- if parsed_script.has_mutation() or query.select_as_cta:
+ if not stopped and (parsed_script.has_mutation() or query.select_as_cta):
conn.commit() # pylint: disable=consider-using-transaction
Review Comment:
Follow-up: my earlier reply in this thread overstated the fix. fbbeb9f178 only covered the success path and the `backend.set` returns-False branch. Two things were still wrong:
1. In the Celery path, after a stop lost the race to a committed mutation or CTAS, any later exception (`results_backend.set` raising, payload serialization raising, `SQLLAB_PAYLOAD_MAX_MB` rejecting the results, the connection failing to close) reached `execute_sql_task` -> `_handle_query_error`, which still preserved the stop and ended STOPPED.
2. Legacy `superset/sql_lab.py` `execute_sql_statements` had the same race, and I had left it out of scope. Its post-commit stop check, its results-backend failure branch and its final status write all reported a committed mutation or CTAS as STOPPED.
Fixed in 4b9f9365d49e371f8fbd2f0b86e98983e59d0ed4. Both executors call an `on_commit` callback once `conn.commit()` succeeds. `execute_sql_task` and `get_sql_results` pass `durable=True` to their error handlers in that case, so the handlers skip the stop check and write FAILED. In the legacy success path, the three post-commit checks no longer return STOPPED once the commit happened, so the query ends SUCCESS or FAILED.
Regression tests (each parametrized for mutation and CTAS):
- `tests/unit_tests/sql/execution/test_celery_task.py::test_failure_after_stop_raced_commit_reports_failed_not_stopped`: runs `execute_sql_task` with a real SQLite metadata DB; the stop is committed from an independent session inside `conn.commit()`. Cases: `backend_set_raises`, `serialize_raises`, `payload_too_large`, `connection_close_raises` (8 fail on fbbeb9f178 with status `stopped`), and `backend_write_false` (2 passed already, kept as a control). All 10 pass on 4b9f9365d4.
- `tests/unit_tests/sql_lab_test.py::test_get_sql_results_keeps_committed_work_when_stop_races_commit`: the same setup through `get_sql_results`. Cases: no failure (expects SUCCESS), `backend_write_false`, `backend_set_raises`, `serialize_raises`, `payload_too_large`, `returned_payload_too_large`, `connection_close_raises`. All 14 fail on fbbeb9f178 with status `stopped` and pass on 4b9f9365d4.
Siblings checked:
- superset/sql/execution/celery_task.py:415 — fixed in 4b9f9365d4 (`execute_sql_task` -> `_handle_query_error` for post-commit exceptions)
- superset/sql/execution/celery_task.py:593 — fixed in fbbeb9f178 (post-commit stop re-check; also `_finalize_successful_query` at :164)
- superset/sql/execution/celery_task.py:296 — fixed in fbbeb9f178 (`_store_results_in_backend` write-false branch; the control cases above cover it)
- superset/sql/execution/celery_task.py:618 — fixed in fbbeb9f178 (`_publish_query_result` durable write)
- superset/sql_lab.py:275 — fixed in 4b9f9365d4 (`get_sql_results` -> `handle_query_error` for post-commit exceptions, including the inline-results size check at :883 that runs after the SUCCESS commit)
- superset/sql_lab.py:695 — fixed in 4b9f9365d4 (post-commit stop check)
- superset/sql_lab.py:803 — fixed in 4b9f9365d4 (results-backend write-failure branch)
- superset/sql_lab.py:859 — fixed in 4b9f9365d4 (final SUCCESS write)
- superset/sql_lab.py:650 — not affected: the per-block `handle_query_error` runs before `conn.commit()`, so nothing is committed yet
- superset/sql/execution/executor.py:525 — not affected: `SQLExecutor.execute` does not commit the target connection and has no stop check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4278161301
##########
superset/daos/query.py:
##########
@@ -70,26 +70,50 @@ def stop_query(client_id: str) -> None:
if not query:
raise QueryNotFoundException(f"Query with client_id {client_id} not found")
- if query.status in [
- QueryStatus.FAILED,
- QueryStatus.SUCCESS,
- QueryStatus.TIMED_OUT,
- ]:
+ if query.status in FINISHED_QUERY_STATUSES:
logger.warning(
"Query with client_id could not be stopped: query already complete",
)
return
- if not sql_lab.cancel_query(query):
+ cancelled = sql_lab.cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ # Preparing the stop can observe the worker's terminal commit.
+ logger.warning(
+ "Query with client_id could not be stopped: query already complete",
+ )
+ return
+ if not cancelled:
raise SupersetCancelQueryException("Could not cancel query")
+ if query.status == QueryStatus.STOPPED:
+ # The engine spec already committed the stop while preparing it
+ # (see ImpalaEngineSpec.prepare_cancel_query).
+ return
+
# cancel_query() may have staged an early-cancel flag on query.extra
# without committing it (see its docstring/comments); the
# @transaction decorator commits it together with status=STOPPED
# below in one transaction, closing the window where another
# request could observe the flag set but the status still RUNNING.
- query.status = QueryStatus.STOPPED
- query.end_time = now_as_float()
+ #
+ # The write is conditional: the worker can commit a terminal status
+ # after the check above, and that result must not be overwritten.
+ stopped = (
+ db.session.query(Query)
+ .filter(
+ Query.id == query.id,
+ Query.status.notin_(FINISHED_QUERY_STATUSES),
Review Comment:
Correction to the "Remaining" note in my previous reply: this PR now does change celery_task._handle_query_error. Commit 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c keeps a committed STOPPED when the worker's error handler runs after the cancel (tests/unit_tests/sql/test_async_cancel.py::test_worker_error_after_committed_stop_keeps_stopped fails without it), so the async cancel chain ends STOPPED with this PR alone. https://github.com/apache/superset/pull/45283 changes the same function; the two will be aligned before either merges.
Remaining: the success write and the results-backend FAILED write in celery_task.py can still overwrite a STOPPED committed after the worker's last stop check. That finish-versus-stop race is the same on master and is handled in https://github.com/apache/superset/pull/45283.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4276923245
##########
superset/sql/execution/executor.py:
##########
@@ -1450,33 +1451,60 @@ def _cancel_query(database: Database, query: Any) -> bool:
:param query: Query model instance to cancel
:returns: True if cancelled successfully, False otherwise
"""
+ from superset.common.db_query_status import FINISHED_QUERY_STATUSES, QueryStatus
from superset.constants import QUERY_CANCEL_KEY, QUERY_EARLY_CANCEL_KEY
+ from superset.models.sql_lab import Query as QueryModel
from superset.utils.core import QuerySource
+ from superset.utils.dates import now_as_float
- # Some engines implicitly handle cancellation
- if database.db_engine_spec.has_implicit_cancel():
- return True
-
- # Some databases may need to make preparations for query cancellation
- database.db_engine_spec.prepare_cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ return False
- # Check early cancellation flag
- if query.extra.get(QUERY_EARLY_CANCEL_KEY):
- return True
+ cancelled = database.db_engine_spec.has_implicit_cancel()
+ if not cancelled:
+ database.db_engine_spec.prepare_cancel_query(query)
+ # The query may have finished while the stop was being prepared.
+ if query.status in FINISHED_QUERY_STATUSES:
+ return False
+ cancelled = bool(query.extra.get(QUERY_EARLY_CANCEL_KEY))
+
+ if not cancelled:
+ cancel_query_id = query.extra.get(QUERY_CANCEL_KEY)
+ if cancel_query_id is None:
+ return False
+
+ with database.get_sqla_engine(
+ catalog=query.catalog,
+ schema=query.schema,
+ source=QuerySource.SQL_LAB,
+ ) as engine:
+ with contextlib.closing(engine.raw_connection()) as conn:
+ with contextlib.closing(conn.cursor()) as cursor:
+ cancelled = database.db_engine_spec.cancel_query(
+ cursor, query, cancel_query_id
+ )
- # Get cancel ID
- cancel_query_id = query.extra.get(QUERY_CANCEL_KEY)
- if cancel_query_id is None:
+ if not cancelled:
return False
+ if query.status == QueryStatus.STOPPED:
+ # Impala commits its stop while preparing cursor-side cancellation.
+ return True
- # Execute cancellation
- with database.get_sqla_engine(
- catalog=query.catalog,
- schema=query.schema,
- source=QuerySource.SQL_LAB,
- ) as engine:
- with contextlib.closing(engine.raw_connection()) as conn:
- with contextlib.closing(conn.cursor()) as cursor:
- return database.db_engine_spec.cancel_query(
- cursor, query, cancel_query_id
- )
+ # Match QueryDAO.stop_query: preserve SUCCESS/TIMED_OUT committed while
+ # cancellation was in flight, but replace a FAILED caused by the kill.
+ stopped = (
+ db.session.query(QueryModel)
+ .filter(
+ QueryModel.id == query.id,
+ QueryModel.status.notin_((QueryStatus.SUCCESS, QueryStatus.TIMED_OUT)),
Review Comment:
This commits STOPPED and returns True, but the Celery worker's error path still ends the run as FAILED. `_handle_query_error` (celery_task.py:88-89) only preserves TIMED_OUT, so when the killed statement raises after this commit (Trino's `_cancel_requested_query` kill, Hive/Presto after `cursor.cancel()`, or Postgres/MySQL when the worker commits last), the row flips STOPPED to FAILED even though `cancel()` reported success. The new docs and the Trino docstring both say the error handler keeps the stop. Should `_handle_query_error` refresh and keep STOPPED the way `sql_lab.handle_query_error` does, with a test that raises after the stop?
##########
superset/sql/execution/celery_task.py:
##########
@@ -481,15 +484,23 @@ def _execute_sql_statements(
)
) from ex
- # Check if stopped
- if not execution_results:
- payload.update({"status": QueryStatus.STOPPED.value})
- return payload
+ # Cancelling can clear cursor.description, leaving a nonempty result
+ # list that looks like successful DML. Check intent, not just results.
+ stopped = not execution_results or check_stopped()
# Commit for mutations
- if parsed_script.has_mutation() or query.select_as_cta:
+ if not stopped and (parsed_script.has_mutation() or query.select_as_cta):
Review Comment:
The new `not stopped` guard on the mutation/CTAS commit isn't exercised by any test. The added Celery stop tests return a SELECT with `select_as_cta=False`, so `has_mutation()` is false and `conn.commit()` is never reached; removing this guard leaves them green, while a stopped UPDATE or CTAS would then be committed. Could one case use an UPDATE or CTAS script with stop intent seen during execution and assert `conn.commit` is not called?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4288033686
##########
superset/daos/query.py:
##########
@@ -70,26 +70,50 @@ def stop_query(client_id: str) -> None:
if not query:
raise QueryNotFoundException(f"Query with client_id {client_id} not found")
- if query.status in [
- QueryStatus.FAILED,
- QueryStatus.SUCCESS,
- QueryStatus.TIMED_OUT,
- ]:
+ if query.status in FINISHED_QUERY_STATUSES:
logger.warning(
"Query with client_id could not be stopped: query already complete",
)
return
- if not sql_lab.cancel_query(query):
+ cancelled = sql_lab.cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ # Preparing the stop can observe the worker's terminal commit.
+ logger.warning(
+ "Query with client_id could not be stopped: query already complete",
+ )
+ return
+ if not cancelled:
raise SupersetCancelQueryException("Could not cancel query")
+ if query.status == QueryStatus.STOPPED:
+ # The engine spec already committed the stop while preparing it
+ # (see ImpalaEngineSpec.prepare_cancel_query).
+ return
+
# cancel_query() may have staged an early-cancel flag on query.extra
# without committing it (see its docstring/comments); the
# @transaction decorator commits it together with status=STOPPED
# below in one transaction, closing the window where another
# request could observe the flag set but the status still RUNNING.
- query.status = QueryStatus.STOPPED
- query.end_time = now_as_float()
+ #
+ # The write is conditional: the worker can commit a terminal status
+ # after the check above, and that result must not be overwritten.
+ stopped = (
+ db.session.query(Query)
+ .filter(
+ Query.id == query.id,
+ Query.status.notin_(FINISHED_QUERY_STATUSES),
Review Comment:
Remaining: generic async cancellation persistence and Celery error/stop finalization are outside this PR and belong to https://github.com/apache/superset/pull/45283 . My earlier async-persistence and error-handler follow-ups above are superseded by the scope reduction requested in https://github.com/apache/superset/pull/45232#pullrequestreview-5537157969 . The synchronous stop claim and Trino follow-up remain in this PR. Current head: daf2d551660983edb8a8f38f150fdb3619dbc4a4.
##########
superset/sql/execution/executor.py:
##########
@@ -1450,33 +1451,60 @@ def _cancel_query(database: Database, query: Any) -> bool:
:param query: Query model instance to cancel
:returns: True if cancelled successfully, False otherwise
"""
+ from superset.common.db_query_status import FINISHED_QUERY_STATUSES, QueryStatus
from superset.constants import QUERY_CANCEL_KEY, QUERY_EARLY_CANCEL_KEY
+ from superset.models.sql_lab import Query as QueryModel
from superset.utils.core import QuerySource
+ from superset.utils.dates import now_as_float
- # Some engines implicitly handle cancellation
- if database.db_engine_spec.has_implicit_cancel():
- return True
-
- # Some databases may need to make preparations for query cancellation
- database.db_engine_spec.prepare_cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ return False
- # Check early cancellation flag
- if query.extra.get(QUERY_EARLY_CANCEL_KEY):
- return True
+ cancelled = database.db_engine_spec.has_implicit_cancel()
+ if not cancelled:
+ database.db_engine_spec.prepare_cancel_query(query)
+ # The query may have finished while the stop was being prepared.
+ if query.status in FINISHED_QUERY_STATUSES:
+ return False
+ cancelled = bool(query.extra.get(QUERY_EARLY_CANCEL_KEY))
+
+ if not cancelled:
+ cancel_query_id = query.extra.get(QUERY_CANCEL_KEY)
+ if cancel_query_id is None:
+ return False
+
+ with database.get_sqla_engine(
+ catalog=query.catalog,
+ schema=query.schema,
+ source=QuerySource.SQL_LAB,
+ ) as engine:
+ with contextlib.closing(engine.raw_connection()) as conn:
+ with contextlib.closing(conn.cursor()) as cursor:
+ cancelled = database.db_engine_spec.cancel_query(
+ cursor, query, cancel_query_id
+ )
- # Get cancel ID
- cancel_query_id = query.extra.get(QUERY_CANCEL_KEY)
- if cancel_query_id is None:
+ if not cancelled:
return False
+ if query.status == QueryStatus.STOPPED:
+ # Impala commits its stop while preparing cursor-side cancellation.
+ return True
- # Execute cancellation
- with database.get_sqla_engine(
- catalog=query.catalog,
- schema=query.schema,
- source=QuerySource.SQL_LAB,
- ) as engine:
- with contextlib.closing(engine.raw_connection()) as conn:
- with contextlib.closing(conn.cursor()) as cursor:
- return database.db_engine_spec.cancel_query(
- cursor, query, cancel_query_id
- )
+ # Match QueryDAO.stop_query: preserve SUCCESS/TIMED_OUT committed while
+ # cancellation was in flight, but replace a FAILED caused by the kill.
+ stopped = (
+ db.session.query(QueryModel)
+ .filter(
+ QueryModel.id == query.id,
+ QueryModel.status.notin_((QueryStatus.SUCCESS, QueryStatus.TIMED_OUT)),
Review Comment:
Remaining: generic async cancellation persistence and Celery error/stop finalization are outside this PR and belong to https://github.com/apache/superset/pull/45283 . My earlier async-persistence and error-handler follow-ups above are superseded by the scope reduction requested in https://github.com/apache/superset/pull/45232#pullrequestreview-5537157969 . The synchronous stop claim and Trino follow-up remain in this PR. Current head: daf2d551660983edb8a8f38f150fdb3619dbc4a4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6136909280
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/45232?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `33.33333%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.08%. Comparing base ([`1cedab0`](https://app.codecov.io/gh/apache/superset/commit/1cedab01e9e2513e1c7548590a451c6434987b52?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c1f5e58`](https://app.codecov.io/gh/apache/superset/commit/c1f5e58cf3c09a7cc2557035ebd88fc52791adf9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/45232?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/db\_engine\_specs/impala.py](https://app.codecov.io/gh/apache/superset/pull/45232?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Fimpala.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2ltcGFsYS5weQ==) | 33.33% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/45232?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #45232 +/- ##
===========================================
- Coverage 83.08% 65.08% -18.00%
===========================================
Files 3028 3028
Lines 196549 196548 -1
Branches 45899 45898 -1
===========================================
- Hits 163310 127931 -35379
- Misses 30015 65732 +35717
+ Partials 3224 2885 -339
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `35.06% <33.33%> (+<0.01%)` | :arrow_up: |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.32% <33.33%> (-0.01%)` | :arrow_down: |
| [presto](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `36.84% <33.33%> (+<0.01%)` | :arrow_up: |
| [python](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.40% <33.33%> (-33.31%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/45232/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/45232?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6138115242
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | fe8719e77d286ad596cb8893178df103e09f0d7f |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6acdb049e75ab727943689d2 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-45232--superset-docs-preview.netlify.app](https://deploy-preview-45232--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcoxDoAgDADAv3S2iUEF8Qc-o1ASTZA2ioMx_l22G-4FPeXQCgtAB-GkErfm_VDKhFcVxSK41arIcoecMLaScsucNMuzcuuWIod-9MlNFJxxfhzs7NnA9wM) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-45232--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6161844459
## CodeAnt PR Risk: Medium Risk
- The PR needs attention before merging.
- A Stop can arrive after the status check but before the mutation or CTAS commit, so changes may be committed even though the query is reported as stopped.
**Assessed commit:** `ce10e4f06817`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4278969897
##########
superset/sql/execution/celery_task.py:
##########
@@ -481,25 +544,44 @@ def _execute_sql_statements(
)
) from ex
- # Check if stopped
- if not execution_results:
- payload.update({"status": QueryStatus.STOPPED.value})
- return payload
+ # Cancelling can clear cursor.description, leaving a nonempty result
+ # list that looks like successful DML. Check intent, not just results.
+ stopped = not execution_results or check_stopped()
# Commit for mutations
- if parsed_script.has_mutation() or query.select_as_cta:
+ if not stopped and (parsed_script.has_mutation() or query.select_as_cta):
conn.commit() # pylint: disable=consider-using-transaction
Review Comment:
Correction: my earlier replies in this thread are wrong for this PR. When I split it, the all-engine stop/commit finalization work (the `on_commit` callback, the `durable=True` error handling and the legacy `superset/sql_lab.py` changes) moved to https://github.com/apache/superset/pull/45283 . At this PR's head 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c that code is not present, so this PR keeps the Impala routing change and the finished-status checks it needs.
Remaining: a stop that loses the race to a committed mutation or CTAS can still end as STOPPED on the Celery and legacy SQL Lab paths (success and post-commit failure). That work is in https://github.com/apache/superset/pull/45283 , not here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4275787664
##########
superset/daos/query.py:
##########
@@ -70,26 +70,50 @@ def stop_query(client_id: str) -> None:
if not query:
raise QueryNotFoundException(f"Query with client_id {client_id} not found")
- if query.status in [
- QueryStatus.FAILED,
- QueryStatus.SUCCESS,
- QueryStatus.TIMED_OUT,
- ]:
+ if query.status in FINISHED_QUERY_STATUSES:
logger.warning(
"Query with client_id could not be stopped: query already complete",
)
return
- if not sql_lab.cancel_query(query):
+ cancelled = sql_lab.cancel_query(query)
+ if query.status in FINISHED_QUERY_STATUSES:
+ # Preparing the stop can observe the worker's terminal commit.
+ logger.warning(
+ "Query with client_id could not be stopped: query already complete",
+ )
+ return
+ if not cancelled:
raise SupersetCancelQueryException("Could not cancel query")
+ if query.status == QueryStatus.STOPPED:
+ # The engine spec already committed the stop while preparing it
+ # (see ImpalaEngineSpec.prepare_cancel_query).
+ return
+
# cancel_query() may have staged an early-cancel flag on query.extra
# without committing it (see its docstring/comments); the
# @transaction decorator commits it together with status=STOPPED
# below in one transaction, closing the window where another
# request could observe the flag set but the status still RUNNING.
- query.status = QueryStatus.STOPPED
- query.end_time = now_as_float()
+ #
+ # The write is conditional: the worker can commit a terminal status
+ # after the check above, and that result must not be overwritten.
+ stopped = (
+ db.session.query(Query)
+ .filter(
+ Query.id == query.id,
+ Query.status.notin_(FINISHED_QUERY_STATUSES),
Review Comment:
Agreed, the conditional write was too broad. Fixed in 8d0fcf8ab3f09d29b2b21afbd120cac688f18c19: `QueryDAO.stop_query` now skips the STOPPED write only for SUCCESS and TIMED_OUT, so a FAILED committed by the worker after the cancel killed its statement is replaced with STOPPED again, as on master.
Regression test, added to `tests/unit_tests/dao/query_stop_isolation_test.py` as you suggested: `test_stop_query_replaces_failure_caused_by_its_cancel`. The worker commits FAILED ("terminating connection due to administrator command") inside `cancel_query`, on the real SQLite harness. At 70fec12f it fails with `AssertionError: assert 'failed' == 'stopped'`; at 8d0fcf8ab3 it passes. `test_stop_query_keeps_outcome_a_stop_cannot_cause[success|timed_out]` covers the statuses that stay protected.
Siblings checked:
- superset/daos/query.py:109 — fixed in 8d0fcf8ab3f09d29b2b21afbd120cac688f18c19
- superset/daos/query.py:80 — not affected: for engines that kill from outside, status is only in-memory there and nothing refreshes it after the kill
- superset/db_engine_specs/impala.py:283 — not affected: the claim runs before any cancel reaches Impala, so a FAILED it skips was not caused by the stop
- superset/sql_lab.py:891 — not affected: Trino's `prepare_cancel_query` commit runs before the kill, so only a FAILED that predates the stop is seen
- superset/sql/execution/executor.py:1465 — not affected: same reason as `sql_lab.py:891`
- superset/sql_lab.py:136 — not affected: `handle_query_error` reloads status and keeps a committed STOPPED
- superset/sql_lab.py:828 — not affected: the SUCCESS write keeps FAILED and STOPPED
- superset/db_engine_specs/hive.py:447 — not affected: cancels from the worker and writes no status (Presto `presto.py:1430` and Trino `trino.py:375` are the same)
Remaining (same on master, not changed by this PR): a stop through `SQLExecutor._cancel_query` (`superset/sql/execution/executor.py:1441`) never writes STOPPED, and `celery_task._handle_query_error` (`superset/sql/execution/celery_task.py:88`) writes FAILED, so on an engine that kills from outside that path can end FAILED. Nothing in the repo calls `AsyncQueryHandle.cancel()`; I left it out of this PR.
Tests: `query_stop_isolation_test.py`, `queries_test.py`, `test_impala_pending.py`, `test_executor.py`, `test_celery_task.py`, `sql_lab_test.py`: 395 passed. The only failures are 8 `test_apply_limit_generate_series_returned_rows` cases that need `duckdb`, which isn't installed locally; they fail the same way at 70fec12f.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #45232:
URL: https://github.com/apache/superset/pull/45232#discussion_r4278081965
##########
superset/sql/execution/celery_task.py:
##########
@@ -481,15 +484,23 @@ def _execute_sql_statements(
)
) from ex
- # Check if stopped
- if not execution_results:
- payload.update({"status": QueryStatus.STOPPED.value})
- return payload
+ # Cancelling can clear cursor.description, leaving a nonempty result
+ # list that looks like successful DML. Check intent, not just results.
+ stopped = not execution_results or check_stopped()
# Commit for mutations
- if parsed_script.has_mutation() or query.select_as_cta:
+ if not stopped and (parsed_script.has_mutation() or query.select_as_cta):
Review Comment:
Remaining: the sync SQL Lab path (superset/sql_lab.py:632) still commits a stopped UPDATE/CTAS when the stop lands after the last block; an independent check at 758e87f60fa44f24e8d7297aa61a4cfb3d4bf28c confirmed that. This PR does not change that path. The fix (re-read the status just before the commit and skip it when a stop is recorded) is in https://github.com/apache/superset/pull/45283 at commit dd5670cc8185700a68384ee8171ae0940c64a20c, with tests/unit_tests/sql_lab_test.py::test_get_sql_results_stop_before_target_commit_stays_stopped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6143358347
Re-reviewed the full PR diff against master at `30afc2b0ecbb98e66ec3542d4c496221f74fd09f`, after verifying the live head.
- **MEDIUM — Celery terminal metadata could still overwrite a committed stop.** A stop landing after the final status read but before the success/error/backend-failure commit could become SUCCESS or FAILED. Lazy relationship loads could also autoflush stale terminal metadata before the final check, overwriting the recorded stop time or status. Closed by **447eb1e57283d724e4fd9a68a3e72a7958e29ef3**: conditionally update the non-STOPPED row before flushing terminal metadata, suppress premature autoflush, and reload the winning stop with a locking read. Five regression cases using separate SQLAlchemy sessions failed before their fixes; successful pending metadata remains preserved.
Verified the earlier fixes: atomic Impala STOPPED/early-cancel publication, running and admission-queued live-cursor cancellation without HTTP double-cancel, cancelled cursors not finalized as SUCCESS, late serialization/backend stops, and the stacktrace cancellation-metadata fixture. Chart/MCP HTTP cancellation remains unchanged. No other findings in the reviewed diff.
Validation: **623 focused tests passed**; **2,330 SQL tests passed with 100% statement/branch coverage**. Pre-commit, including mypy, passed on all five PR-changed files. Real-Impala acceptance was not run.
Fix pushed normally to the designated fork branch. Verified live head `447eb1e57283d724e4fd9a68a3e72a7958e29ef3`; the PR remains draft with base master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6142272605
CI diagnosis at `24e6e6b69b`: **PR-caused test-fixture failure**, not flaky/infra.
- Checked the prior SQL coverage issue first. In https://github.com/apache/superset/actions/runs/38351128036/job/115143663607 the full suite failed before the SQL coverage gate, so that gate was skipped. The only failure was `test_new_handle_query_error_payload_includes_stacktrace_when_enabled`: 26,328 passed, one failed.
- Latest master run https://github.com/apache/superset/actions/runs/38361381834 skipped Python tests. Latest master run that actually executed them, https://github.com/apache/superset/actions/runs/38357477723/job/115165985995 (`6d974e19c5`), passed 26,337 tests. Its JUnit artifact confirms all six stacktrace tests passed; SQL coverage was 100%.
- Reproduced the failure locally: one failed, five passed. The new cancellation check reads `query.extra`, but the stacktrace fixture left it as a truthy `MagicMock`. That falsely selected STOPPED instead of the error/stacktrace path. Set the fixture metadata to `{}` and assert FAILED for both stacktrace settings. No production behavior changed.
- Validation: 106 focused stacktrace/Impala/Celery tests passed; the full SQL gate passed 2,325 tests with 100% statement/branch coverage, including Celery. Pre-commit, including mypy, passed on all five branch-changed files using the repository-pinned Ruff version.
Pushed `30afc2b0ecbb98e66ec3542d4c496221f74fd09f` to the designated fork branch; replacement CI is pending. The PR remains draft. No CI rerun or review request was made. Real-Impala acceptance remains outside this unit-test fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(impala): stop SQL Lab queries through the live cursor only [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #45232:
URL: https://github.com/apache/superset/pull/45232#issuecomment-6138874004
Reviewed fe8719e7 against master after verifying the live head and green CI.
- **MEDIUM — `superset/sql/execution/celery_task.py:512` (reviewed head):** A stop during result serialization or backend storage could still be overwritten by SUCCESS; a backend failure or exception could overwrite STOPPED with FAILED. Four regression cases failed before the fix. Closed by **24e6e6b69b**: re-check late stops, return a minimal STOPPED payload, preserve the recorded end time, and avoid flushing stale state or discarding normal pending result metadata.
Validation: **624 tests passed** across Impala, cancellation validation, SQL Lab, SQL execution/Celery, Hive, Presto, chart cancellation, and MCP workers. Celery statement/branch coverage is **100%**. Pre-commit, including mypy, passed on all four branch-changed files. Added real SQLAlchemy checks for metadata preservation and timeout handling.
Checked running/admission-queued live-cursor cancellation, stop/fetch boundaries, both earlier fixes, and unchanged chart/MCP HTTP cancellation. No other reproducible findings in the reviewed changes. Real-Impala acceptance was not run.
Fix pushed to the designated fork branch. The PR remains draft with base master; CI is running at 24e6e6b69b.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org