You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "blarghmatey (via GitHub)" <gi...@apache.org> on 2026/06/01 17:09:56 UTC
[I] StreamingCSVExportCommand fails for chart CSV exports (Trino + others): __STREAM_ERROR__:Export failed when row_limit >= CSV_STREAMING_ROW_THRESHOLD [superset]
blarghmatey opened a new issue, #40583:
URL: https://github.com/apache/superset/issues/40583
## Description
Chart CSV exports fail with `__STREAM_ERROR__:Export failed` for any chart whose `row_limit` meets or exceeds `CSV_STREAMING_ROW_THRESHOLD` (default: 100,000), when the underlying database is Trino (and likely others).
## Environment
- Superset version: **6.1.0**
- Database backend: **Trino**
- `impersonate_user`: disabled on the database connection
- `GLOBAL_ASYNC_QUERIES`: disabled
- Chart type: **table** viz, `query_mode: raw`, `server_pagination: false`
- Chart `row_limit`: **100,000** (equal to default `CSV_STREAMING_ROW_THRESHOLD`)
## Steps to Reproduce
1. Create or open a table chart with `row_limit` ≥ `CSV_STREAMING_ROW_THRESHOLD` (default 100,000) backed by a Trino database
2. Click **Download → .csv** in the Explore or Dashboard view
3. Observe the download fails; the file contains `__STREAM_ERROR__:Export failed. Please try again in some time.`
4. The Superset pod logs show the caught exception (logged at ERROR level by the generator)
## Root Cause Analysis
### Trigger: `_should_use_streaming` threshold check
In `superset/charts/data/api.py`, `_should_use_streaming()` gates the new `StreamingCSVExportCommand` path:
```python
threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
# Falls back to row_limit when actual rowcount is unavailable
# (non-paginated table charts don't populate queries[1])
actual_row_count = int(row_limit) if row_limit else 0
return actual_row_count is not None and actual_row_count >= threshold
```
For a non-paginated table chart (`server_pagination: false`), the rowcount is not in the second query slot, so the code falls back to `row_limit`. A chart with `row_limit == 100000` therefore **always** triggers the streaming path.
### Bug: generator pushes a new app context over the active request context
`BaseStreamingCSVExportCommand.run()` returns a callable that executes inside a **new** app context when iterated by Flask during response streaming:
```python
def csv_generator() -> Generator[str, None, None]:
with self._current_app.app_context(): # <-- pushes NEW app context
with preserve_g_context(captured_g):
try:
yield from self._execute_query_and_stream(sql, database, limit)
except Exception as e:
logger.error("Error in streaming CSV generator: %s", e)
yield "__STREAM_ERROR__:Export failed. Please try again in some time.\n"
```
`preserve_g_context` restores `g.__dict__` from the original request, but the new app context creates a fresh Flask-SQLAlchemy scoped session scope. Inside `_execute_query_and_stream`:
```python
with db.session() as session:
merged_database = session.merge(database) # database from original session
with merged_database.get_sqla_engine() as engine:
with engine.connect() as connection:
result_proxy = connection.execution_options(
stream_results=True
).execute(text(sql))
```
The combination of (1) a fresh scoped session in the nested app context merging an object loaded in the original request session, (2) `stream_results=True` passed to the Trino SQLAlchemy driver (which may not support server-side cursors in the expected way), and (3) any `DB_CONNECTION_MUTATOR` or `get_sqla_engine()` behaviour relying on request-scoped state produces an exception that is silently swallowed and returned to the client as `__STREAM_ERROR__`.
**The actual traceback is logged at ERROR level** by the generator but the error message that reaches the user provides no debugging information.
## Additional Observations
- The streaming path **re-runs the Trino query from scratch** even though `ChartDataCommand.run()` already fetched the full result set moments earlier. For charts at the threshold boundary (e.g., `row_limit == 100000`), data is fetched twice.
- The non-streaming in-memory `CsvResponse` path works correctly for the same chart, user, and dataset.
- `can_export_streaming_csv` appears in the `ab_permission` table of the Superset metadata DB but **does not exist anywhere in the 6.1.0 Python codebase** and is never checked by any endpoint. This phantom permission causes confusion when diagnosing access errors.
## Workaround
Set `CSV_STREAMING_ROW_THRESHOLD` above the maximum practical `row_limit` in `superset_config.py`:
```python
# Workaround: disable streaming path until validated against all DB drivers
CSV_STREAMING_ROW_THRESHOLD = 1_000_000
```
## Suggested Fixes
1. **Remove the nested app context push.** The generator already runs within the active request context during Flask response streaming; a second app context is unnecessary and causes scoped session conflicts. Use the existing request-scoped `g` and session directly.
2. **Avoid re-fetching data.** `_should_use_streaming` receives the full `result` dict already loaded by `ChartDataCommand.run()`. The streaming path could stream from that in-memory result rather than re-issuing a new database query.
3. **Surface the actual exception** in the error marker or at minimum include a request/trace ID, rather than a generic message with no debugging context.
4. **Fix the threshold comparison**: use `>` instead of `>=` so that a chart with `row_limit` exactly equal to `CSV_STREAMING_ROW_THRESHOLD` uses the proven in-memory path. Or document that the threshold must be set *above* the maximum expected `row_limit`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] StreamingCSVExportCommand fails for chart CSV exports (Trino + others): __STREAM_ERROR__:Export failed when row_limit >= CSV_STREAMING_ROW_THRESHOLD [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #40583:
URL: https://github.com/apache/superset/issues/40583#issuecomment-4748192053
Thanks for the thorough analysis! This is the same streaming-export regression already tracked in #40465 (raw SQL bypassing the standard query-preparation pipeline, breaking Trino and impersonation). Let's keep the discussion consolidated there. Closing this as a duplicate, but please comment on #40465 if you have additional repro details to add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] StreamingCSVExportCommand fails for chart CSV exports (Trino + others): __STREAM_ERROR__:Export failed when row_limit >= CSV_STREAMING_ROW_THRESHOLD [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #40583:
URL: https://github.com/apache/superset/issues/40583#issuecomment-4595021879
<!-- Answer -->
This is a confirmed issue that matches a detailed bug report in [#40465](https://github.com/apache/superset/issues/40465#issuecomment-4554592705) [[1]](https://github.com/apache/superset/issues/40465#issuecomment-4554592705), which identifies two root causes in the streaming CSV export path introduced by PR #35478:
1. **No `mutate_sql_based_on_config()` call** — the streaming path sends raw SQL (including trailing semicolons) directly via `connection.execute(text(sql))`, bypassing the standard pipeline that strips semicolons and applies sqlglot reformatting. Trino rejects trailing semicolons with `SYNTAX_ERROR`.
2. **No user context on engine acquisition** — the streaming path calls `get_sqla_engine()` without passing `user_name`, so impersonation headers are never set and queries run as the service principal.
A partial fix landed in PR #40194 [[2]](https://github.com/apache/superset/pull/40194) to handle PostgreSQL prequeries on streaming connections, but it did **not** address SQL mutation or user impersonation — so the Trino failure and the context issues you've identified remain unresolved.
Your workaround of raising `CSV_STREAMING_ROW_THRESHOLD` above the maximum practical `row_limit` is the correct short-term mitigation. Your suggested fixes align well with the analysis in #40465 — particularly routing the streaming path through the existing `mutate_sql_based_on_config()` pipeline and passing user context to the engine.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fsuperset).*
---
Share context across your team and agents. Try [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-tagline&utm_term=apache%2Fsuperset).
[](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-feedback&utm_term=apache%2Fsuperset&message_id=e8e6ce23-7966-439a-80f1-2036df48c4b4) [](https://github.dosu.com/apache/superset?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-learn-repo&utm_term=apache%2Fsuperset) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-add-team&utm_term=apache%2Fsuperset)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] StreamingCSVExportCommand fails for chart CSV exports (Trino + others): __STREAM_ERROR__:Export failed when row_limit >= CSV_STREAMING_ROW_THRESHOLD [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #40583: StreamingCSVExportCommand fails for chart CSV exports (Trino + others): __STREAM_ERROR__:Export failed when row_limit >= CSV_STREAMING_ROW_THRESHOLD
URL: https://github.com/apache/superset/issues/40583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org