You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "amaannawab923 (via GitHub)" <gi...@apache.org> on 2025/10/03 08:36:19 UTC

[PR] (feat-streaming): Streaming Csv Uploads for over 100k records [superset]

amaannawab923 opened a new pull request, #35478:
URL: https://github.com/apache/superset/pull/35478

   <!---
   Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
   Example:
   fix(dashboard): load charts correctly
   -->
   
   ### SUMMARY
   <!--- Describe the change below, including rationale and design decisions -->
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   ### TESTING INSTRUCTIONS
   <!--- Required! What steps can be taken to manually verify the changes? -->
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2535575478


##########
superset/charts/data/api.py:
##########
@@ -507,3 +549,105 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count: int | None = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    rowcount = data[0].get("rowcount")
+                    actual_row_count = int(rowcount) if rowcount else None
+
+        # Fallback to row_limit if actual count not available
+        if actual_row_count is None:
+            if form_data and "row_limit" in form_data:
+                row_limit = form_data.get("row_limit", 0)
+                actual_row_count = int(row_limit) if row_limit else 0
+            elif query_context.form_data and "row_limit" in query_context.form_data:
+                row_limit = query_context.form_data.get("row_limit", 0)
+                actual_row_count = int(row_limit) if row_limit else 0
+
+        # Use streaming if row count meets or exceeds threshold
+        if actual_row_count is not None and actual_row_count >= threshold:
+            return True
+
+        return False
+
+    def _create_streaming_csv_response(
+        self,
+        result: dict[Any, Any],
+        form_data: dict[str, Any] | None = None,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large datasets."""
+        query_context = result["query_context"]
+
+        # Use filename from frontend if provided, otherwise generate one
+        if not filename:
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            chart_name = "export"
+
+            if form_data and form_data.get("slice_name"):
+                chart_name = form_data["slice_name"]
+            elif form_data and form_data.get("viz_type"):
+                chart_name = form_data["viz_type"]
+
+            # Sanitize chart name for filename
+            filename = secure_filename(f"superset_{chart_name}_{timestamp}.csv")
+
+        logger.info(
+            "Creating streaming CSV response: %s (from frontend: %s)",
+            filename,
+            filename is not None,

Review Comment:
   `filename` can never be `None`, since it's either a truthy string or the result from `secure_filename`.



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,381 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { t } from '@superset-ui/core';
+import { styled, useTheme } from '@apache-superset/core/ui';
+import {
+  Modal,
+  Button,
+  Typography,
+  Progress,
+} from '@superset-ui/core/components';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+const MAX_PROGRESS_PERCENT = 99;
+const COMPLETED_PERCENT = 100;
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  onDownload?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  ${({ theme }) => `
+    padding: ${theme.sizeUnit * 4}px 0 ${theme.sizeUnit * 2}px;
+  `}
+`;
+
+const ProgressSection = styled.div`
+  ${({ theme }) => `
+    margin: ${theme.sizeUnit * 6}px 0;
+    position: relative;
+  `}
+`;
+
+const ProgressWrapper = styled.div`
+  ${({ theme }) => `
+    display: flex;
+    align-items: center;
+    gap: ${theme.sizeUnit * 3}px;
+  `}
+`;
+
+const StyledProgress = styled(Progress)`
+  flex: 1;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  ${({ theme }) => `
+    color: ${theme.colorSuccess};
+    font-size: ${theme.sizeUnit * 6}px;
+    flex-shrink: 0;
+  `}
+`;
+
+const ErrorIconWrapper = styled.div`
+  ${({ theme }) => `
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    width: ${theme.sizeUnit * 4}px;
+    height: ${theme.sizeUnit * 4}px;
+    background-color: ${theme.colorError};
+    border-radius: 50%;
+    flex-shrink: 0;
+  `}
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  ${({ theme }) => `
+    color: ${theme.colorWhite};
+    font-size: ${theme.sizeUnit * 2.5}px;
+  `}
+`;
+
+const ActionButtons = styled.div`
+  ${({ theme }) => `
+    display: flex;
+    gap: ${theme.sizeUnit * 2}px;
+    justify-content: flex-end;
+  `}
+`;
+
+const CenteredText = styled(Text)`
+  ${({ theme }) => `
+    display: block;
+    text-align: center;
+    margin-top: ${theme.sizeUnit * 4}px;
+  `}
+`;
+
+const ErrorText = styled(CenteredText)`
+  ${({ theme }) => `
+    color: ${theme.colorError};
+  `}
+`;
+
+const CancelButton = styled(Button)`
+  ${({ theme }) => `
+    background-color: ${theme.colorSuccessBg};
+    color: ${theme.colorSuccess};
+    border-color: ${theme.colorSuccessBg};
+
+    &:hover {
+      background-color: ${theme.colorSuccessBg};
+      color: ${theme.colorSuccess};
+      border-color: ${theme.colorSuccess};
+    }
+
+    &:focus {
+      background-color: ${theme.colorSuccessBg};
+      color: ${theme.colorSuccess};
+      border-color: ${theme.colorSuccess};
+    }
+  `}
+`;
+
+const DownloadButton = styled(Button)`
+  ${({ theme }) => `
+    background-color: ${theme.colorSuccess};
+    border-color: ${theme.colorSuccess};
+    color: ${theme.colorWhite};
+
+    &:hover:not(:disabled) {
+      background-color: ${theme.colorSuccessActive};
+      border-color: ${theme.colorSuccessActive};
+      color: ${theme.colorWhite};
+    }
+
+    &:focus:not(:disabled) {
+      background-color: ${theme.colorSuccess};
+      border-color: ${theme.colorSuccess};
+      color: ${theme.colorWhite};
+    }
+
+    &:disabled {
+      background-color: ${theme.colorBgContainerDisabled};
+      border-color: ${theme.colorBgContainerDisabled};
+      color: ${theme.colorTextDisabled};
+    }
+  `}
+`;
+
+const triggerFileDownload = (url: string, filename: string) => {
+  const link = document.createElement('a');
+  link.href = url;
+  link.download = filename;
+  document.body.appendChild(link);
+  link.click();
+  document.body.removeChild(link);
+};
+
+const calculateProgressPercentage = (
+  status: ExportStatus,
+  totalRows?: number,
+  rowsProcessed?: number,
+): number => {
+  if (status === ExportStatus.COMPLETED) return COMPLETED_PERCENT;
+
+  if (!totalRows || totalRows <= 0 || !rowsProcessed) return 0;
+
+  const percentage = (rowsProcessed / totalRows) * 100;
+  return Math.round(Math.min(MAX_PROGRESS_PERCENT, percentage));

Review Comment:
   Seems like you don't want to show 100% when the progress is, say, 99.7%. There's an easier way to do this that doesn't rely on `MAX_PROGRESS_PERCENT`:
   
   ```suggestion
     return Math.floor(percentage);
   ```



##########
superset/charts/data/api.py:
##########
@@ -467,10 +486,33 @@ def _get_data_response(
         except ChartDataQueryFailedError as exc:
             return self.response_400(message=exc.message)
 
-        # Log is_cached if extra payload callback is provided
-        self._log_is_cached(result, add_extra_log_payload)
+            # Log is_cached if extra payload callback is provided
+        if add_extra_log_payload and result and "queries" in result:
+            is_cached_values = [query.get("is_cached") for query in result["queries"]]
+            if len(is_cached_values) == 1:
+                add_extra_log_payload(is_cached=is_cached_values[0])
+            elif is_cached_values:
+                add_extra_log_payload(is_cached=is_cached_values)

Review Comment:
   I would just always log a list here:
   
   ```suggestion
               add_extra_log_payload(is_cached=is_cached_values)
   ```



##########
superset/commands/streaming_export/base.py:
##########
@@ -0,0 +1,219 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Base command for streaming CSV exports."""
+
+from __future__ import annotations
+
+import csv
+import io
+import logging
+import time
+from abc import abstractmethod
+from typing import Any, Callable, Generator
+
+from flask import current_app as app
+from sqlalchemy import text
+
+from superset import db
+from superset.commands.base import BaseCommand
+
+logger = logging.getLogger(__name__)
+
+
+class BaseStreamingCSVExportCommand(BaseCommand):
+    """
+    Base class for streaming CSV export commands.
+
+    Provides shared functionality for:
+    - Generating CSV data in chunks
+    - Managing database connections
+    - Buffering data for efficient streaming
+    - Error handling with user-friendly messages
+
+    Subclasses must implement:
+    - _get_sql_and_database(): Return SQL query string and database object
+    - _get_row_limit(): Return optional row limit for the export
+    """
+
+    def __init__(self, chunk_size: int = 1000):
+        """
+        Initialize the streaming export command.
+
+        Args:
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        self._chunk_size = chunk_size
+        self._current_app = app._get_current_object()
+
+    @abstractmethod
+    def _get_sql_and_database(self) -> tuple[str, Any]:
+        """
+        Get the SQL query and database for execution.
+
+        Returns:
+            Tuple of (sql_query, database_object)
+        """
+
+    @abstractmethod
+    def _get_row_limit(self) -> int | None:
+        """
+        Get the row limit for the export.
+
+        Returns:
+            Row limit or None for unlimited
+        """
+
+    def _write_csv_header(
+        self, columns: list[str], csv_writer: Any, buffer: io.StringIO
+    ) -> tuple[str, int]:
+        """Write CSV header and return header data with byte count."""
+        csv_writer.writerow(columns)
+        header_data = buffer.getvalue()
+        total_bytes = len(header_data.encode("utf-8"))
+        buffer.seek(0)
+        buffer.truncate()
+        return header_data, total_bytes
+
+    def _process_rows(
+        self,
+        result_proxy: Any,
+        csv_writer: Any,
+        buffer: io.StringIO,
+        limit: int | None,
+    ) -> Generator[tuple[str, int, int], None, None]:
+        """
+        Process database rows and yield CSV data chunks.
+
+        Yields tuples of (data_chunk, row_count, byte_count).
+        """
+        row_count = 0
+        flush_threshold = 65536  # 64KB
+
+        while rows := result_proxy.fetchmany(self._chunk_size):
+            for row in rows:
+                # Apply limit if specified
+                if limit is not None and row_count >= limit:
+                    break
+
+                csv_writer.writerow(row)
+                row_count += 1
+
+                # Check buffer size and flush if needed
+                current_size = buffer.tell()
+                if current_size >= flush_threshold:
+                    data = buffer.getvalue()
+                    data_bytes = len(data.encode("utf-8"))
+                    yield data, row_count, data_bytes
+                    buffer.seek(0)
+                    buffer.truncate()
+
+            # Break outer loop if limit reached
+            if limit is not None and row_count >= limit:
+                break
+
+        # Flush remaining buffer
+        if remaining_data := buffer.getvalue():
+            data_bytes = len(remaining_data.encode("utf-8"))
+            yield remaining_data, row_count, data_bytes
+
+    def _execute_query_and_stream(
+        self, sql: str, database: Any, limit: int | None
+    ) -> Generator[str, None, None]:
+        """Execute query with streaming and yield CSV chunks."""
+        start_time = time.time()
+        total_bytes = 0
+
+        with db.session() as session:
+            # Merge database to prevent DetachedInstanceError
+            merged_database = session.merge(database)
+
+            # Execute query with streaming
+            with merged_database.get_sqla_engine() as engine:
+                connection = engine.connect()

Review Comment:
   You might be able to run this as a context manager, so you don't need the try/finally block to close it.



##########
superset/sqllab/api.py:
##########
@@ -294,6 +299,119 @@ def export_csv(self, client_id: str) -> CsvResponse:
         )
         return response
 
+    @expose("/export_streaming/", methods=("POST",))
+    @protect()
+    @permission_name("read")
+    @statsd_metrics
+    @event_logger.log_this_with_context(
+        action=lambda self,
+        *args,
+        **kwargs: f"{self.__class__.__name__}.export_streaming_csv",
+        log_to_statsd=False,
+    )
+    def export_streaming_csv(self) -> Response:
+        """Export SQL query results using streaming for large datasets.
+        ---
+        post:
+          summary: Export SQL query results to CSV with streaming
+          requestBody:
+            description: Export parameters
+            required: true
+            content:
+              application/x-www-form-urlencoded:
+                schema:
+                  type: object
+                  properties:
+                    client_id:
+                      type: string
+                      description: The SQL query result identifier
+                    filename:
+                      type: string
+                      description: Optional filename for the export
+                    expected_rows:
+                      type: integer
+                      description: Optional expected row count for progress tracking
+          responses:
+            200:
+              description: Streaming CSV export
+              content:
+                text/csv:
+                  schema:
+                    type: string
+            400:
+              $ref: '#/components/responses/400'
+            401:
+              $ref: '#/components/responses/401'
+            403:
+              $ref: '#/components/responses/403'
+            404:
+              $ref: '#/components/responses/404'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        # Extract parameters from form data
+        client_id = request.form.get("client_id")
+        filename = request.form.get("filename")
+
+        if not client_id:
+            return self.response_400(message="client_id is required")
+
+        expected_rows = None
+        if expected_rows_str := request.form.get("expected_rows"):
+            try:
+                expected_rows = int(expected_rows_str)
+            except (ValueError, TypeError):
+                logger.warning("Invalid expected_rows value: %s", expected_rows_str)
+
+        return self._create_streaming_csv_response(client_id, filename, expected_rows)
+
+    def _create_streaming_csv_response(
+        self,
+        client_id: str,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large SQL Lab result sets."""
+        # Execute streaming command
+        chunk_size = 1000
+        command = StreamingSqlResultExportCommand(client_id, chunk_size)
+        command.validate()
+
+        if not filename:
+            query = command._query
+            assert query is not None

Review Comment:
   This is not used?
   
   ```suggestion
   ```



##########
superset/sqllab/api.py:
##########
@@ -294,6 +299,119 @@ def export_csv(self, client_id: str) -> CsvResponse:
         )
         return response
 
+    @expose("/export_streaming/", methods=("POST",))
+    @protect()
+    @permission_name("read")
+    @statsd_metrics
+    @event_logger.log_this_with_context(
+        action=lambda self,
+        *args,
+        **kwargs: f"{self.__class__.__name__}.export_streaming_csv",
+        log_to_statsd=False,
+    )
+    def export_streaming_csv(self) -> Response:
+        """Export SQL query results using streaming for large datasets.
+        ---
+        post:
+          summary: Export SQL query results to CSV with streaming
+          requestBody:
+            description: Export parameters
+            required: true
+            content:
+              application/x-www-form-urlencoded:
+                schema:
+                  type: object
+                  properties:
+                    client_id:
+                      type: string
+                      description: The SQL query result identifier
+                    filename:
+                      type: string
+                      description: Optional filename for the export
+                    expected_rows:
+                      type: integer
+                      description: Optional expected row count for progress tracking
+          responses:
+            200:
+              description: Streaming CSV export
+              content:
+                text/csv:
+                  schema:
+                    type: string
+            400:
+              $ref: '#/components/responses/400'
+            401:
+              $ref: '#/components/responses/401'
+            403:
+              $ref: '#/components/responses/403'
+            404:
+              $ref: '#/components/responses/404'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        # Extract parameters from form data
+        client_id = request.form.get("client_id")
+        filename = request.form.get("filename")
+
+        if not client_id:
+            return self.response_400(message="client_id is required")
+
+        expected_rows = None
+        if expected_rows_str := request.form.get("expected_rows"):
+            try:
+                expected_rows = int(expected_rows_str)
+            except (ValueError, TypeError):
+                logger.warning("Invalid expected_rows value: %s", expected_rows_str)
+
+        return self._create_streaming_csv_response(client_id, filename, expected_rows)
+
+    def _create_streaming_csv_response(
+        self,
+        client_id: str,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large SQL Lab result sets."""
+        # Execute streaming command
+        chunk_size = 1000

Review Comment:
   I see this hardcoded as 1000 everywhere, we might want to make this configurable (we can do that in the future). I wonder if 1024 would be better due to page size alignment, eg.



##########
superset/commands/sql_lab/streaming_export_command.py:
##########
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of SQL Lab query results."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask_babel import gettext as __
+
+from superset import db
+from superset.commands.streaming_export.base import BaseStreamingCSVExportCommand
+from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
+from superset.exceptions import SupersetErrorException, SupersetSecurityException
+from superset.models.sql_lab import Query
+from superset.sql.parse import SQLScript
+from superset.sqllab.limiting_factor import LimitingFactor
+
+
+class StreamingSqlResultExportCommand(BaseStreamingCSVExportCommand):
+    """
+    Command to execute a streaming CSV export of SQL Lab query results.
+
+    This command handles SQL Lab-specific logic:
+    - Query validation and access control
+    - SQL parsing and limit extraction
+    - LimitingFactor-based row limit adjustment
+    """
+
+    def __init__(
+        self,
+        client_id: str,
+        chunk_size: int = 1000,
+    ):
+        """
+        Initialize the SQL Lab streaming export command.
+
+        Args:
+            client_id: The SQL Lab query client ID
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        super().__init__(chunk_size)
+        self._client_id = client_id
+        self._query: Query | None = None
+
+    def validate(self) -> None:
+        """Validate permissions and query existence."""
+        self._query = (
+            db.session.query(Query).filter_by(client_id=self._client_id).one_or_none()
+        )
+        if self._query is None:
+            raise SupersetErrorException(
+                SupersetError(
+                    message=__(
+                        "The query associated with these results could not be found. "
+                        "You need to re-run the original query."
+                    ),
+                    error_type=SupersetErrorType.RESULTS_BACKEND_ERROR,
+                    level=ErrorLevel.ERROR,
+                ),
+                status=404,
+            )
+
+        try:
+            self._query.raise_for_access()
+        except SupersetSecurityException as ex:
+            raise SupersetErrorException(
+                SupersetError(
+                    message=__("Cannot access the query"),
+                    error_type=SupersetErrorType.QUERY_SECURITY_ACCESS_ERROR,
+                    level=ErrorLevel.ERROR,
+                ),
+                status=403,
+            ) from ex
+
+    def _get_sql_and_database(self) -> tuple[str, Any]:
+        """
+        Get the SQL query and database for SQL Lab export.
+
+        Returns:
+            Tuple of (sql_query, database_object)
+        """
+        assert self._query is not None
+
+        select_sql = self._query.select_sql
+        executed_sql = self._query.executed_sql
+        database = self._query.database
+
+        # Get the SQL query
+        if select_sql:
+            sql = select_sql
+        else:
+            sql = executed_sql

Review Comment:
   More idiomatic:
   
   
   ```suggestion
           sql = select_sql or executed_sql
   ```



##########
superset/charts/data/api.py:
##########
@@ -507,3 +549,105 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count: int | None = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    rowcount = data[0].get("rowcount")
+                    actual_row_count = int(rowcount) if rowcount else None
+
+        # Fallback to row_limit if actual count not available
+        if actual_row_count is None:
+            if form_data and "row_limit" in form_data:
+                row_limit = form_data.get("row_limit", 0)
+                actual_row_count = int(row_limit) if row_limit else 0
+            elif query_context.form_data and "row_limit" in query_context.form_data:
+                row_limit = query_context.form_data.get("row_limit", 0)
+                actual_row_count = int(row_limit) if row_limit else 0
+
+        # Use streaming if row count meets or exceeds threshold
+        if actual_row_count is not None and actual_row_count >= threshold:
+            return True
+
+        return False

Review Comment:
   ```suggestion
           # Use streaming if row count meets or exceeds threshold
           return actual_row_count is not None and actual_row_count >= threshold
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "geido (via GitHub)" <gi...@apache.org>.
geido merged PR #35478:
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3386379464

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/18380946650) for [f9a72c8](https://github.com/apache/superset/commit/f9a72c8)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] (feat-streaming): Streaming Csv Uploads for over 100k records [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3364785452

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3511558827

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19232767048) for [3fba394](https://github.com/apache/superset/commit/3fba394)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3527338499

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19329778273) for [ed5c1e3](https://github.com/apache/superset/commit/ed5c1e3)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3503567056

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/19174467104) for [3fba394](https://github.com/apache/superset/commit/3fba394)
   
   β€’ **Environment:** http://52.26.102.225:8080 (admin/admin)
   β€’ **Lifetime:** 48h auto-cleanup
   β€’ **Updates:** New commits create fresh environments automatically


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3527338494

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19329778354) for [ed5c1e3](https://github.com/apache/superset/commit/ed5c1e3)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3383294624

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/18358486742) for [c6d1838](https://github.com/apache/superset/commit/c6d1838)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2410737632


##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,215 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+  exportType: 'csv' | 'xlsx';

Review Comment:
   ### Unused exportType prop <sub>![category Design](https://img.shields.io/badge/Design-0d9488)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   The exportType prop is defined in the interface but never used in the component implementation.
   
   
   ###### Why this matters
   This creates a disconnect between the component's interface and its actual behavior, potentially misleading developers about the component's capabilities and making it harder to extend for different export types.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Either remove the unused prop from the interface or implement logic to handle different export types:
   
   ```typescript
   // Option 1: Remove unused prop
   interface StreamingExportModalProps {
     visible: boolean;
     onCancel: () => void;
     onRetry?: () => void;
     progress: StreamingProgress;
   }
   
   // Option 2: Use the prop in the component
   const getDefaultFilename = () => {
     return exportType === 'xlsx' ? 'export.xlsx' : 'export.csv';
   };
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/2d07a5cb-d66f-4468-a9b4-b08c17670e53/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/2d07a5cb-d66f-4468-a9b4-b08c17670e53?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/2d07a5cb-d66f-4468-a9b4-b08c17670e53?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/2d07a5cb-d66f-4468-a9b4-b08c17670e53?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/2d07a5cb-d66f-4468-a9b4-b08c17670e53)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:cec4fa32-c247-4ced-808c-ae1673228a40 -->
   
   
   [](cec4fa32-c247-4ced-808c-ae1673228a40)



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,215 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+  exportType: 'csv' | 'xlsx';
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)``;
+
+const DownloadButton = styled(Button)``;
+
+const StreamingExportModal = ({
+  visible,
+  onCancel,
+  onRetry,
+  progress,
+}: StreamingExportModalProps) => {
+  const { status, downloadUrl, filename, error } = progress;
+
+  const getProgressPercentage = (): number => {
+    if (status === ExportStatus.COMPLETED) return 100;
+    if (progress.totalRows && progress.totalRows > 0) {
+      const percentage = Math.min(
+        99,
+        (progress.rowsProcessed / progress.totalRows) * 100,
+      );
+      return Math.round(percentage);
+    }
+    return 0;
+  };
+
+  const handleDownload = () => {
+    if (downloadUrl) {
+      const link = document.createElement('a');
+      link.href = downloadUrl;
+      link.download = filename || 'export.csv';
+      document.body.appendChild(link);
+      link.click();
+      document.body.removeChild(link);
+      onCancel();
+    }
+  };
+
+  let content;
+  if (status === ExportStatus.ERROR) {
+    content = (
+      <ModalContent>
+        <ProgressSection>
+          <Progress percent={0} status="exception" showInfo={false} />
+          <ErrorText type="danger">{error || t('Export failed')}</ErrorText>
+        </ProgressSection>
+        <ActionButtons>
+          <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+          {onRetry && (
+            <DownloadButton type="primary" onClick={onRetry}>
+              {t('Retry')}
+            </DownloadButton>
+          )}
+        </ActionButtons>
+      </ModalContent>
+    );
+  } else if (status === ExportStatus.CANCELLED) {
+    content = (
+      <ModalContent>
+        <ProgressSection>
+          <Progress
+            percent={getProgressPercentage()}
+            status="exception"
+            showInfo={false}
+          />
+          <ProgressText>{t('Export cancelled')}</ProgressText>
+        </ProgressSection>
+        <ActionButtons>
+          <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+          {onRetry && (
+            <DownloadButton type="primary" onClick={onRetry}>
+              {t('Retry')}
+            </DownloadButton>
+          )}
+        </ActionButtons>
+      </ModalContent>
+    );
+  } else if (status === ExportStatus.COMPLETED) {
+    content = (
+      <ModalContent>
+        <ProgressSection>
+          <Progress percent={100} status="success" showInfo={false} />
+          <ProgressText>
+            {t('Export successful: %s', filename || 'export.csv')}
+          </ProgressText>
+        </ProgressSection>
+        <ActionButtons>
+          <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+          <DownloadButton
+            type="primary"
+            onClick={handleDownload}
+            disabled={!downloadUrl}
+          >
+            {t('Download')}
+          </DownloadButton>
+        </ActionButtons>
+      </ModalContent>
+    );
+  } else {
+    content = (
+      <ModalContent>
+        <ProgressSection>
+          <Progress
+            percent={getProgressPercentage()}
+            status="active"
+            showInfo
+            format={percent => `${Math.round(percent || 0)}%`}
+          />
+          <ProgressText>
+            {filename
+              ? t('Processing export for %s', filename)
+              : t('Processing export...')}
+          </ProgressText>
+        </ProgressSection>
+        <ActionButtons>
+          <CancelButton onClick={onCancel}>{t('Cancel')}</CancelButton>
+          <DownloadButton type="primary" disabled>
+            {t('Download')}
+          </DownloadButton>
+        </ActionButtons>
+      </ModalContent>
+    );
+  }

Review Comment:
   ### Modal content rendering needs componentization <sub>![category Design](https://img.shields.io/badge/Design-0d9488)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   The modal content rendering logic uses a large if-else block with significant code duplication across different status conditions.
   
   
   ###### Why this matters
   This approach makes the code harder to maintain and extend. Each status branch contains similar structure with duplicated components, making it difficult to make consistent changes across all states.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Extract common modal content structure into reusable components and use a status-based mapping:
   ```typescript
   const StatusContent = {
     [ExportStatus.ERROR]: ({ error, onCancel, onRetry }) => (
       <ModalContent>
         <ProgressSection>
           <Progress percent={0} status="exception" showInfo={false} />
           <ErrorText type="danger">{error || t('Export failed')}</ErrorText>
         </ProgressSection>
         <ActionButtons {...{ onCancel, onRetry }} />
       </ModalContent>
     ),
     // ... similar for other statuses
   };
   
   return <Modal>{StatusContent[status](props)}</Modal>;
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d1e76e1e-962e-4bbe-beca-cb3eb809bff5/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d1e76e1e-962e-4bbe-beca-cb3eb809bff5?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d1e76e1e-962e-4bbe-beca-cb3eb809bff5?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d1e76e1e-962e-4bbe-beca-cb3eb809bff5?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d1e76e1e-962e-4bbe-beca-cb3eb809bff5)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:b60d57f5-2d8f-4d47-a9b3-7add3e6402be -->
   
   
   [](b60d57f5-2d8f-4d47-a9b3-7add3e6402be)



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef } from 'react';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: any;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}
+
+export const useStreamingExport = (options: UseStreamingExportOptions = {}) => {
+  const [progress, setProgress] = useState<StreamingProgress>({
+    rowsProcessed: 0,
+    totalRows: undefined,
+    totalSize: 0,
+    speed: 0,
+    mbPerSecond: 0,
+    elapsedTime: 0,
+    status: ExportStatus.STREAMING,
+  });
+  const [isExporting, setIsExporting] = useState(false);
+  const abortControllerRef = useRef<AbortController | null>(null);
+
+  const updateProgress = useCallback((updates: Partial<StreamingProgress>) => {
+    setProgress(prev => ({ ...prev, ...updates }));
+  }, []);
+
+  const startExport = useCallback(
+    async ({
+      url,
+      payload,
+      filename,
+      exportType,
+      expectedRows,
+    }: StreamingExportParams) => {
+      if (isExporting) return;
+
+      setIsExporting(true);
+      abortControllerRef.current = new AbortController();
+
+      updateProgress({
+        rowsProcessed: 0,
+        totalRows: expectedRows,
+        totalSize: 0,
+        speed: 0,
+        mbPerSecond: 0,
+        elapsedTime: 0,
+        status: ExportStatus.STREAMING,
+        filename,
+      });
+
+      try {
+        const response = await fetch(url, {
+          method: 'POST',
+          headers: {
+            'Content-Type': 'application/x-www-form-urlencoded',
+          },
+          body: new URLSearchParams({
+            form_data: JSON.stringify(payload),
+            filename: filename || `export.${exportType}`,
+            expected_rows: expectedRows?.toString() || '',
+          }),
+          signal: abortControllerRef.current.signal,
+          credentials: 'same-origin',
+        });
+
+        if (!response.ok) {
+          throw new Error(
+            `Export failed: ${response.status} ${response.statusText}`,
+          );

Review Comment:
   ### Large function with multiple responsibilities <sub>![category Design](https://img.shields.io/badge/Design-0d9488)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   The startExport function is too large and handles multiple responsibilities including request setup, streaming, progress tracking, and error handling.
   
   
   ###### Why this matters
   Large functions with multiple responsibilities are harder to maintain, test, and understand. They violate the Single Responsibility Principle and make the code less modular.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Extract the streaming logic into separate functions:
   ```typescript
   const handleStreamingResponse = async (reader: ReadableStreamDefaultReader, expectedRows: number) => {
     // Handle streaming logic
   };
   
   const createDownloadBlob = (data: Uint8Array, exportType: 'csv' | 'xlsx') => {
     // Handle blob creation
   };
   
   const startExport = useCallback(async (params: StreamingExportParams) => {
     const response = await setupExportRequest(params);
     const streamedData = await handleStreamingResponse(response.body.getReader(), params.expectedRows);
     const downloadUrl = createDownloadBlob(streamedData, params.exportType);
     handleExportCompletion(downloadUrl, params.filename);
   }, []);
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/319bab71-831e-472e-a958-adab45c2a2f2/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/319bab71-831e-472e-a958-adab45c2a2f2?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/319bab71-831e-472e-a958-adab45c2a2f2?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/319bab71-831e-472e-a958-adab45c2a2f2?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/319bab71-831e-472e-a958-adab45c2a2f2)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:85bca7c6-01a8-4209-bbdb-fe0928f4fbc6 -->
   
   
   [](85bca7c6-01a8-4209-bbdb-fe0928f4fbc6)



##########
superset/commands/chart/data/streaming_export_command.py:
##########
@@ -0,0 +1,169 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of large datasets."""
+
+from __future__ import annotations
+
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+
+from superset.commands.base import BaseCommand
+
+if TYPE_CHECKING:
+    from superset.common.query_context import QueryContext
+
+logger = logging.getLogger(__name__)
+
+
+class StreamingCSVExportCommand(BaseCommand):
+    """
+    Command to execute a streaming CSV export.
+
+    This command handles the business logic for:
+    - Executing database queries with server-side cursors
+    - Generating CSV data in chunks
+    - Managing database connections
+    - Buffering data for efficient streaming
+    """
+
+    def __init__(
+        self,
+        query_context: QueryContext,
+        chunk_size: int = 1000,
+    ):
+        """
+        Initialize the streaming export command.
+
+        Args:
+            query_context: The query context containing datasource and query details
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        self._query_context = query_context
+        self._chunk_size = chunk_size
+        self._current_app = app._get_current_object()
+
+    def validate(self) -> None:
+        """Validate permissions and query context."""
+        self._query_context.raise_for_access()
+
+    def run(self) -> Callable[[], Generator[str, None, None]]:
+        """
+        Execute the streaming CSV export.
+
+        Returns:
+            A callable that returns a generator yielding CSV data chunks as strings.
+            The callable is needed to maintain Flask app context during streaming.
+        """
+
+        def csv_generator() -> Generator[str, None, None]:
+            """Generator that yields CSV data from database query."""
+            with self._current_app.app_context():
+                start_time = time.time()
+                total_bytes = 0
+
+                try:
+                    from superset import db
+                    from superset.connectors.sqla.models import SqlaTable
+
+                    datasource = self._query_context.datasource
+
+                    with db.session() as session:
+                        if isinstance(datasource, SqlaTable):
+                            datasource = session.merge(datasource)
+
+                        query_obj = self._query_context.queries[0]
+                        sql_query = datasource.get_query_str(query_obj.to_dict())
+
+                        with datasource.database.get_sqla_engine() as engine:
+                            connection = engine.connect()
+
+                            try:
+                                from sqlalchemy import text
+
+                                result_proxy = connection.execution_options(
+                                    stream_results=True
+                                ).execute(text(sql_query))
+
+                                columns = list(result_proxy.keys())
+
+                                # Yield CSV header
+                                header_row = (
+                                    ",".join(f'"{col}"' for col in columns) + "\n"
+                                )
+                                total_bytes += len(header_row.encode("utf-8"))
+                                yield header_row
+
+                                row_count = 0
+                                buffer = []
+                                buffer_size = 0
+                                flush_threshold = 65536  # 64KB
+
+                                while True:
+                                    rows = result_proxy.fetchmany(self._chunk_size)
+                                    if not rows:
+                                        break
+
+                                    for row in rows:
+                                        csv_row = ",".join(
+                                            f'"{str(cell) if cell is not None else ""}"'
+                                            for cell in row
+                                        )

Review Comment:
   ### Improper CSV quote escaping <sub>![category Functionality](https://img.shields.io/badge/Functionality-0284c7)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   CSV escaping is incorrect - quotes within cell values are not properly escaped, breaking CSV format.
   
   
   ###### Why this matters
   If cell values contain quotes, the generated CSV will be malformed and unparseable by CSV readers, causing data corruption or import failures.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Use proper CSV escaping by doubling internal quotes:
   ```python
   def escape_csv_value(cell):
       if cell is None:
           return '""'
       cell_str = str(cell)
       # Escape quotes by doubling them
       escaped = cell_str.replace('"', '""')
       return f'"{escaped}"'
   
   csv_row = ",".join(escape_csv_value(cell) for cell in row)
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/a7fd517a-698a-4d8c-8adc-ed53022104db/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/a7fd517a-698a-4d8c-8adc-ed53022104db?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/a7fd517a-698a-4d8c-8adc-ed53022104db?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/a7fd517a-698a-4d8c-8adc-ed53022104db?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/a7fd517a-698a-4d8c-8adc-ed53022104db)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:7a0fde4b-d907-4916-8e02-b7aa93e782ce -->
   
   
   [](7a0fde4b-d907-4916-8e02-b7aa93e782ce)



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,215 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+  exportType: 'csv' | 'xlsx';
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)``;
+
+const DownloadButton = styled(Button)``;
+
+const StreamingExportModal = ({
+  visible,
+  onCancel,
+  onRetry,
+  progress,
+}: StreamingExportModalProps) => {
+  const { status, downloadUrl, filename, error } = progress;
+
+  const getProgressPercentage = (): number => {
+    if (status === ExportStatus.COMPLETED) return 100;
+    if (progress.totalRows && progress.totalRows > 0) {
+      const percentage = Math.min(
+        99,
+        (progress.rowsProcessed / progress.totalRows) * 100,
+      );
+      return Math.round(percentage);
+    }
+    return 0;
+  };
+
+  const handleDownload = () => {
+    if (downloadUrl) {
+      const link = document.createElement('a');
+      link.href = downloadUrl;
+      link.download = filename || 'export.csv';

Review Comment:
   ### Hardcoded CSV extension in fallback filename <sub>![category Functionality](https://img.shields.io/badge/Functionality-0284c7)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   The hardcoded fallback filename 'export.csv' doesn't respect the exportType prop, always defaulting to CSV extension regardless of the actual export format.
   
   
   ###### Why this matters
   This could cause confusion for users when downloading XLSX files that get a .csv extension, potentially leading to file opening issues or user confusion about the actual file format.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Use the exportType prop to determine the correct file extension:
   
   ```typescript
   const getDefaultFilename = () => {
     return exportType === 'xlsx' ? 'export.xlsx' : 'export.csv';
   };
   
   const handleDownload = () => {
     if (downloadUrl) {
       const link = document.createElement('a');
       link.href = downloadUrl;
       link.download = filename || getDefaultFilename();
       document.body.appendChild(link);
       link.click();
       document.body.removeChild(link);
       onCancel();
     }
   };
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/fe3a567d-c40b-42c0-ba75-2a2022617d24/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/fe3a567d-c40b-42c0-ba75-2a2022617d24?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/fe3a567d-c40b-42c0-ba75-2a2022617d24?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/fe3a567d-c40b-42c0-ba75-2a2022617d24?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/fe3a567d-c40b-42c0-ba75-2a2022617d24)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:140f2a20-a124-42cc-aed8-d2289e90a10c -->
   
   
   [](140f2a20-a124-42cc-aed8-d2289e90a10c)



##########
superset/commands/chart/data/streaming_export_command.py:
##########
@@ -0,0 +1,169 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of large datasets."""
+
+from __future__ import annotations
+
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+
+from superset.commands.base import BaseCommand
+
+if TYPE_CHECKING:
+    from superset.common.query_context import QueryContext
+
+logger = logging.getLogger(__name__)
+
+
+class StreamingCSVExportCommand(BaseCommand):
+    """
+    Command to execute a streaming CSV export.
+
+    This command handles the business logic for:
+    - Executing database queries with server-side cursors
+    - Generating CSV data in chunks
+    - Managing database connections
+    - Buffering data for efficient streaming
+    """
+
+    def __init__(
+        self,
+        query_context: QueryContext,
+        chunk_size: int = 1000,
+    ):
+        """
+        Initialize the streaming export command.
+
+        Args:
+            query_context: The query context containing datasource and query details
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        self._query_context = query_context
+        self._chunk_size = chunk_size
+        self._current_app = app._get_current_object()
+
+    def validate(self) -> None:
+        """Validate permissions and query context."""
+        self._query_context.raise_for_access()
+
+    def run(self) -> Callable[[], Generator[str, None, None]]:
+        """
+        Execute the streaming CSV export.
+
+        Returns:
+            A callable that returns a generator yielding CSV data chunks as strings.
+            The callable is needed to maintain Flask app context during streaming.
+        """
+
+        def csv_generator() -> Generator[str, None, None]:
+            """Generator that yields CSV data from database query."""
+            with self._current_app.app_context():
+                start_time = time.time()
+                total_bytes = 0
+
+                try:
+                    from superset import db
+                    from superset.connectors.sqla.models import SqlaTable
+
+                    datasource = self._query_context.datasource
+
+                    with db.session() as session:
+                        if isinstance(datasource, SqlaTable):
+                            datasource = session.merge(datasource)
+
+                        query_obj = self._query_context.queries[0]
+                        sql_query = datasource.get_query_str(query_obj.to_dict())
+
+                        with datasource.database.get_sqla_engine() as engine:
+                            connection = engine.connect()
+
+                            try:
+                                from sqlalchemy import text
+
+                                result_proxy = connection.execution_options(
+                                    stream_results=True
+                                ).execute(text(sql_query))
+
+                                columns = list(result_proxy.keys())
+
+                                # Yield CSV header
+                                header_row = (
+                                    ",".join(f'"{col}"' for col in columns) + "\n"
+                                )
+                                total_bytes += len(header_row.encode("utf-8"))
+                                yield header_row
+
+                                row_count = 0
+                                buffer = []
+                                buffer_size = 0
+                                flush_threshold = 65536  # 64KB
+
+                                while True:
+                                    rows = result_proxy.fetchmany(self._chunk_size)
+                                    if not rows:
+                                        break
+
+                                    for row in rows:
+                                        csv_row = ",".join(
+                                            f'"{str(cell) if cell is not None else ""}"'
+                                            for cell in row
+                                        )
+                                        csv_line = csv_row + "\n"
+                                        row_bytes = len(csv_line.encode("utf-8"))
+                                        total_bytes += row_bytes
+                                        row_count += 1
+
+                                        buffer.append(csv_line)
+                                        buffer_size += row_bytes
+
+                                        if buffer_size >= flush_threshold:
+                                            yield "".join(buffer)

Review Comment:
   ### Inefficient buffer flushing with string concatenation <sub>![category Performance](https://img.shields.io/badge/Performance-4f46e5)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   Using a list buffer with string join for each flush creates O(n) string concatenation overhead when the buffer could be yielded incrementally.
   
   
   ###### Why this matters
   The join operation creates a new string containing all buffered rows, causing memory spikes and CPU overhead that defeats the purpose of streaming, especially when buffer sizes are large.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Use a more efficient buffering strategy or yield individual rows:
   
   ```python
   from io import StringIO
   
   buffer = StringIO()
   for row in rows:
       csv_line = format_csv_row(row)
       buffer.write(csv_line)
       if buffer.tell() >= flush_threshold:
           yield buffer.getvalue()
           buffer = StringIO()
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/8ff8bee7-7507-4a9c-87f5-fb2eb00daa7c/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/8ff8bee7-7507-4a9c-87f5-fb2eb00daa7c?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/8ff8bee7-7507-4a9c-87f5-fb2eb00daa7c?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/8ff8bee7-7507-4a9c-87f5-fb2eb00daa7c?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/8ff8bee7-7507-4a9c-87f5-fb2eb00daa7c)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:a4214305-a879-43e9-9670-a0ae95487d40 -->
   
   
   [](a4214305-a879-43e9-9670-a0ae95487d40)



##########
superset/commands/chart/data/streaming_export_command.py:
##########
@@ -0,0 +1,169 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of large datasets."""
+
+from __future__ import annotations
+
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+
+from superset.commands.base import BaseCommand
+
+if TYPE_CHECKING:
+    from superset.common.query_context import QueryContext
+
+logger = logging.getLogger(__name__)
+
+
+class StreamingCSVExportCommand(BaseCommand):
+    """
+    Command to execute a streaming CSV export.
+
+    This command handles the business logic for:
+    - Executing database queries with server-side cursors
+    - Generating CSV data in chunks
+    - Managing database connections
+    - Buffering data for efficient streaming
+    """
+
+    def __init__(
+        self,
+        query_context: QueryContext,
+        chunk_size: int = 1000,
+    ):
+        """
+        Initialize the streaming export command.
+
+        Args:
+            query_context: The query context containing datasource and query details
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        self._query_context = query_context
+        self._chunk_size = chunk_size
+        self._current_app = app._get_current_object()
+
+    def validate(self) -> None:
+        """Validate permissions and query context."""
+        self._query_context.raise_for_access()
+
+    def run(self) -> Callable[[], Generator[str, None, None]]:
+        """
+        Execute the streaming CSV export.
+
+        Returns:
+            A callable that returns a generator yielding CSV data chunks as strings.
+            The callable is needed to maintain Flask app context during streaming.
+        """
+
+        def csv_generator() -> Generator[str, None, None]:
+            """Generator that yields CSV data from database query."""
+            with self._current_app.app_context():
+                start_time = time.time()
+                total_bytes = 0
+
+                try:
+                    from superset import db
+                    from superset.connectors.sqla.models import SqlaTable
+
+                    datasource = self._query_context.datasource
+
+                    with db.session() as session:
+                        if isinstance(datasource, SqlaTable):
+                            datasource = session.merge(datasource)
+
+                        query_obj = self._query_context.queries[0]
+                        sql_query = datasource.get_query_str(query_obj.to_dict())
+
+                        with datasource.database.get_sqla_engine() as engine:
+                            connection = engine.connect()
+
+                            try:
+                                from sqlalchemy import text
+
+                                result_proxy = connection.execution_options(
+                                    stream_results=True
+                                ).execute(text(sql_query))
+
+                                columns = list(result_proxy.keys())
+
+                                # Yield CSV header
+                                header_row = (
+                                    ",".join(f'"{col}"' for col in columns) + "\n"
+                                )
+                                total_bytes += len(header_row.encode("utf-8"))
+                                yield header_row
+
+                                row_count = 0
+                                buffer = []
+                                buffer_size = 0
+                                flush_threshold = 65536  # 64KB
+
+                                while True:
+                                    rows = result_proxy.fetchmany(self._chunk_size)
+                                    if not rows:
+                                        break
+
+                                    for row in rows:
+                                        csv_row = ",".join(
+                                            f'"{str(cell) if cell is not None else ""}"'
+                                            for cell in row
+                                        )

Review Comment:
   ### Inefficient CSV cell formatting in streaming loop <sub>![category Performance](https://img.shields.io/badge/Performance-4f46e5)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   String concatenation in a tight loop using join() with generator expression performs unnecessary string conversions and allocations for each cell.
   
   
   ###### Why this matters
   For large datasets, this creates significant CPU overhead and memory pressure due to repeated str() calls and string object creation for every cell, potentially negating the memory benefits of streaming.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Use a more efficient approach by pre-allocating a list and avoiding repeated str() conversions:
   
   ```python
   csv_parts = []
   for cell in row:
       if cell is None:
           csv_parts.append('""')
       else:
           csv_parts.append(f'"{cell}"')
   csv_row = ",".join(csv_parts)
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/48613e2a-0dc4-41fb-8797-b1df519bc348/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/48613e2a-0dc4-41fb-8797-b1df519bc348?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/48613e2a-0dc4-41fb-8797-b1df519bc348?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/48613e2a-0dc4-41fb-8797-b1df519bc348?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/48613e2a-0dc4-41fb-8797-b1df519bc348)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:fa58ca17-d4e8-46b2-a874-daead0a25b90 -->
   
   
   [](fa58ca17-d4e8-46b2-a874-daead0a25b90)



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef } from 'react';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: any;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}
+
+export const useStreamingExport = (options: UseStreamingExportOptions = {}) => {
+  const [progress, setProgress] = useState<StreamingProgress>({
+    rowsProcessed: 0,
+    totalRows: undefined,
+    totalSize: 0,
+    speed: 0,
+    mbPerSecond: 0,
+    elapsedTime: 0,
+    status: ExportStatus.STREAMING,
+  });
+  const [isExporting, setIsExporting] = useState(false);
+  const abortControllerRef = useRef<AbortController | null>(null);
+
+  const updateProgress = useCallback((updates: Partial<StreamingProgress>) => {
+    setProgress(prev => ({ ...prev, ...updates }));
+  }, []);
+
+  const startExport = useCallback(
+    async ({
+      url,
+      payload,
+      filename,
+      exportType,
+      expectedRows,
+    }: StreamingExportParams) => {
+      if (isExporting) return;
+
+      setIsExporting(true);
+      abortControllerRef.current = new AbortController();
+
+      updateProgress({
+        rowsProcessed: 0,
+        totalRows: expectedRows,
+        totalSize: 0,
+        speed: 0,
+        mbPerSecond: 0,
+        elapsedTime: 0,
+        status: ExportStatus.STREAMING,
+        filename,
+      });
+
+      try {
+        const response = await fetch(url, {
+          method: 'POST',
+          headers: {
+            'Content-Type': 'application/x-www-form-urlencoded',
+          },
+          body: new URLSearchParams({
+            form_data: JSON.stringify(payload),
+            filename: filename || `export.${exportType}`,
+            expected_rows: expectedRows?.toString() || '',
+          }),
+          signal: abortControllerRef.current.signal,
+          credentials: 'same-origin',
+        });
+
+        if (!response.ok) {
+          throw new Error(
+            `Export failed: ${response.status} ${response.statusText}`,
+          );
+        }
+
+        if (!response.body) {
+          throw new Error('Response body is not available for streaming');
+        }
+
+        const reader = response.body.getReader();
+        const chunks: Uint8Array[] = [];
+        let receivedLength = 0;
+        let rowsProcessed = 0;
+        const NEWLINE_BYTE = 10; // '\n' character code
+
+        // eslint-disable-next-line no-constant-condition
+        while (true) {
+          // eslint-disable-next-line no-await-in-loop
+          const { done, value } = await reader.read();
+
+          if (done) break;
+
+          if (abortControllerRef.current?.signal.aborted) {
+            throw new Error('Export cancelled by user');
+          }
+
+          chunks.push(value);
+          receivedLength += value.length;
+
+          // Count newlines directly in binary (faster than decoding + regex)
+          let newlineCount = 0;
+          for (let i = 0; i < value.length; i += 1) {
+            if (value[i] === NEWLINE_BYTE) {
+              newlineCount += 1;
+            }
+          }

Review Comment:
   ### Inefficient newline counting in large chunks <sub>![category Performance](https://img.shields.io/badge/Performance-4f46e5)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   Byte-by-byte iteration to count newlines is inefficient for large chunks and could be optimized using native array methods.
   
   
   ###### Why this matters
   This manual loop approach is slower than using optimized native methods, adding unnecessary processing overhead that accumulates significantly with large datasets.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Use more efficient methods for counting newlines:
   
   ```typescript
   // More efficient approach using filter
   const newlineCount = value.filter(byte => byte === NEWLINE_BYTE).length;
   
   // Or using reduce for better performance with very large chunks
   const newlineCount = value.reduce((count, byte) => 
     byte === NEWLINE_BYTE ? count + 1 : count, 0
   );
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3c8cd078-3c5a-4aae-9a06-19d6f7cac16d/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3c8cd078-3c5a-4aae-9a06-19d6f7cac16d?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3c8cd078-3c5a-4aae-9a06-19d6f7cac16d?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3c8cd078-3c5a-4aae-9a06-19d6f7cac16d?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3c8cd078-3c5a-4aae-9a06-19d6f7cac16d)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:f5afdd6b-2cb8-4711-8753-a7483405a5b5 -->
   
   
   [](f5afdd6b-2cb8-4711-8753-a7483405a5b5)



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef } from 'react';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: any;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}
+
+export const useStreamingExport = (options: UseStreamingExportOptions = {}) => {
+  const [progress, setProgress] = useState<StreamingProgress>({
+    rowsProcessed: 0,
+    totalRows: undefined,
+    totalSize: 0,
+    speed: 0,
+    mbPerSecond: 0,
+    elapsedTime: 0,
+    status: ExportStatus.STREAMING,
+  });
+  const [isExporting, setIsExporting] = useState(false);
+  const abortControllerRef = useRef<AbortController | null>(null);
+
+  const updateProgress = useCallback((updates: Partial<StreamingProgress>) => {
+    setProgress(prev => ({ ...prev, ...updates }));
+  }, []);
+
+  const startExport = useCallback(
+    async ({
+      url,
+      payload,
+      filename,
+      exportType,
+      expectedRows,
+    }: StreamingExportParams) => {
+      if (isExporting) return;
+
+      setIsExporting(true);
+      abortControllerRef.current = new AbortController();
+
+      updateProgress({
+        rowsProcessed: 0,
+        totalRows: expectedRows,
+        totalSize: 0,
+        speed: 0,
+        mbPerSecond: 0,
+        elapsedTime: 0,
+        status: ExportStatus.STREAMING,
+        filename,
+      });
+
+      try {
+        const response = await fetch(url, {
+          method: 'POST',
+          headers: {
+            'Content-Type': 'application/x-www-form-urlencoded',
+          },
+          body: new URLSearchParams({
+            form_data: JSON.stringify(payload),
+            filename: filename || `export.${exportType}`,
+            expected_rows: expectedRows?.toString() || '',
+          }),
+          signal: abortControllerRef.current.signal,
+          credentials: 'same-origin',
+        });
+
+        if (!response.ok) {
+          throw new Error(
+            `Export failed: ${response.status} ${response.statusText}`,
+          );
+        }
+
+        if (!response.body) {
+          throw new Error('Response body is not available for streaming');
+        }
+
+        const reader = response.body.getReader();
+        const chunks: Uint8Array[] = [];
+        let receivedLength = 0;
+        let rowsProcessed = 0;
+        const NEWLINE_BYTE = 10; // '\n' character code
+
+        // eslint-disable-next-line no-constant-condition
+        while (true) {
+          // eslint-disable-next-line no-await-in-loop
+          const { done, value } = await reader.read();
+
+          if (done) break;
+
+          if (abortControllerRef.current?.signal.aborted) {
+            throw new Error('Export cancelled by user');
+          }
+
+          chunks.push(value);
+          receivedLength += value.length;
+
+          // Count newlines directly in binary (faster than decoding + regex)
+          let newlineCount = 0;
+          for (let i = 0; i < value.length; i += 1) {
+            if (value[i] === NEWLINE_BYTE) {
+              newlineCount += 1;
+            }
+          }
+          rowsProcessed += newlineCount;

Review Comment:
   ### Inaccurate CSV row counting with quoted newlines <sub>![category Functionality](https://img.shields.io/badge/Functionality-0284c7)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   Row counting logic assumes each newline represents a complete row, but CSV data can contain newlines within quoted fields, leading to inaccurate progress reporting.
   
   
   ###### Why this matters
   This will cause the progress indicator to show incorrect row counts when CSV data contains multi-line fields (e.g., text with embedded newlines in quotes), potentially confusing users about export progress and completion status.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Implement proper CSV parsing to count actual rows instead of just newlines. Consider using a streaming CSV parser or maintain state to track whether we're inside quoted fields:
   
   ```typescript
   // Track if we're inside a quoted field
   let insideQuotes = false;
   let newlineCount = 0;
   for (let i = 0; i < value.length; i += 1) {
     const byte = value[i];
     if (byte === 34) { // '"' character
       insideQuotes = !insideQuotes;
     } else if (byte === NEWLINE_BYTE && !insideQuotes) {
       newlineCount += 1;
     }
   }
   rowsProcessed += newlineCount;
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3993c389-5e4b-439b-b4d5-1ca33329eabf/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3993c389-5e4b-439b-b4d5-1ca33329eabf?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3993c389-5e4b-439b-b4d5-1ca33329eabf?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3993c389-5e4b-439b-b4d5-1ca33329eabf?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/3993c389-5e4b-439b-b4d5-1ca33329eabf)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:45307a87-8d86-44b1-8f87-c8d4116697a5 -->
   
   
   [](45307a87-8d86-44b1-8f87-c8d4116697a5)



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef } from 'react';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: any;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}

Review Comment:
   ### Loose typing with 'any' <sub>![category Design](https://img.shields.io/badge/Design-0d9488)</sub>
   
   <details>
     <summary>Tell me more</summary>
   
   ###### What is the issue?
   The payload parameter is typed as 'any', which reduces type safety and makes the interface less predictable.
   
   
   ###### Why this matters
   Using 'any' bypasses TypeScript's type checking system, potentially leading to runtime errors that could have been caught during development.
   
   ###### Suggested change βˆ™ *Feature Preview*
   Define a specific type for the payload:
   ```typescript
   interface ExportPayload {
     // Define specific payload fields
     [key: string]: string | number | boolean | object;
   }
   
   interface StreamingExportParams {
     url: string;
     payload: ExportPayload;
     filename?: string;
     exportType: 'csv' | 'xlsx';
     expectedRows?: number;
   }
   ```
   
   
   ###### Provide feedback to improve future suggestions
   [![Nice Catch](https://img.shields.io/badge/πŸ‘%20Nice%20Catch-71BC78)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/e5468571-46f1-4382-8a00-6c44dd48ce35/upvote) [![Incorrect](https://img.shields.io/badge/πŸ‘Ž%20Incorrect-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/e5468571-46f1-4382-8a00-6c44dd48ce35?what_not_true=true)  [![Not in Scope](https://img.shields.io/badge/πŸ‘Ž%20Out%20of%20PR%20scope-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/e5468571-46f1-4382-8a00-6c44dd48ce35?what_out_of_scope=true) [![Not in coding standard](https://img.shields.io/badge/πŸ‘Ž%20Not%20in%20our%20standards-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/e5468571-46f1-4382-8a00-6c44dd48ce35?what_not_in_standard=true) [![Other](https://img.shields.io/badge/πŸ‘Ž%20Other-white)](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/e5468571-46f1-4382-8a00-6c44dd48ce35)
   </details>
   
   <sub>
   
   πŸ’¬ Looking for more details? Reply to this comment to chat with Korbit.
   </sub>
   
   <!--- korbi internal id:55bb78fb-3708-430a-b846-7fdebbdb30e5 -->
   
   
   [](55bb78fb-3708-430a-b846-7fdebbdb30e5)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "LevisNgigi (via GitHub)" <gi...@apache.org>.
LevisNgigi commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2419425306


##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,403 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/* eslint-disable theme-colors/no-literal-colors */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+  position: relative;
+`;
+
+const ProgressWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  gap: ${({ theme }) => theme.sizeUnit * 3}px;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  color: #52c41a;
+  font-size: 24px;
+  flex-shrink: 0;
+`;
+
+const ErrorIconWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 16px;
+  height: 16px;
+  background-color: #ff4d4f;
+  border-radius: 50%;
+  flex-shrink: 0;
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  color: white;
+  font-size: 10px;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)`
+  background-color: #f0fff8;
+  color: #1c997a;
+  border-color: #f0fff8;
+
+  &:hover {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+
+  &:focus {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+`;
+
+const DownloadButton = styled(Button)`
+  &.ant-btn-primary {
+    background-color: #2ec196;
+    border-color: #2ec196;
+    color: #ffffff;
+
+    &:hover:not(:disabled) {
+      background-color: #26a880;
+      border-color: #26a880;
+      color: #ffffff;
+    }
+
+    &:focus:not(:disabled) {
+      background-color: #2ec196;
+      border-color: #2ec196;
+      color: #ffffff;
+    }
+
+    &:disabled {
+      background-color: #f2f2f2;
+      border-color: #f2f2f2;
+      color: #b5b5b5;
+    }
+  }
+`;
+
+interface ModalStateContentProps {
+  status: ExportStatus;
+  progress: StreamingProgress;
+  onCancel: () => void;
+  onRetry?: () => void;
+  onDownload: () => void;
+  getProgressPercentage: () => number;
+}
+
+const ErrorContent = ({
+  error,
+  onCancel,
+  onRetry,
+  getProgressPercentage,
+}: {
+  error?: string;
+  onCancel: () => void;
+  onRetry?: () => void;
+  getProgressPercentage: () => number;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <ProgressWrapper>
+        <Progress
+          percent={getProgressPercentage()}
+          status="exception"
+          showInfo={false}
+          style={{ flex: 1 }}
+        />
+        <ErrorIconWrapper>
+          <ErrorIconStyled />
+        </ErrorIconWrapper>
+      </ProgressWrapper>
+      <ErrorText type="danger">{error || t('Export failed')}</ErrorText>
+    </ProgressSection>
+    <ActionButtons>
+      <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+      {onRetry && (
+        <DownloadButton type="primary" onClick={onRetry}>
+          {t('Retry')}
+        </DownloadButton>
+      )}
+    </ActionButtons>
+  </ModalContent>
+);
+
+const CancelledContent = ({
+  getProgressPercentage,
+  onCancel,
+  onRetry,
+}: {
+  getProgressPercentage: () => number;
+  onCancel: () => void;
+  onRetry?: () => void;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <Progress
+        percent={getProgressPercentage()}
+        status="exception"
+        showInfo={false}
+      />
+      <ProgressText>{t('Export cancelled')}</ProgressText>
+    </ProgressSection>
+    <ActionButtons>
+      <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+      {onRetry && (
+        <DownloadButton type="primary" onClick={onRetry}>

Review Comment:
   Your DownloadButton already handles primary styling in line 128, might not need type="primary" here



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef, useEffect } from 'react';
+import { SupersetClient } from '@superset-ui/core';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportPayload {
+  [key: string]: unknown;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: StreamingExportPayload;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}
+
+const NEWLINE_BYTE = 10; // '\n' character code
+
+const createFetchRequest = async (
+  _url: string,
+  payload: StreamingExportPayload,
+  filename: string,
+  _exportType: string,
+  expectedRows: number | undefined,
+  signal: AbortSignal,
+): Promise<RequestInit> => {
+  const headers: Record<string, string> = {
+    'Content-Type': 'application/x-www-form-urlencoded',
+  };
+
+  // Get CSRF token using SupersetClient
+  const csrfToken = await SupersetClient.getCSRFToken();
+  if (csrfToken) {
+    headers['X-CSRFToken'] = csrfToken;
+  }
+
+  // Build form data - if payload has client_id, it's SQL Lab export
+  // Otherwise it's a chart export with form_data
+  const formParams: Record<string, string> = {
+    filename,
+    expected_rows: expectedRows?.toString() || '',
+  };
+
+  if ('client_id' in payload) {
+    // SQL Lab export - pass client_id directly
+    formParams.client_id = String(payload.client_id);
+  } else {
+    // Chart export - wrap payload in form_data
+    formParams.form_data = JSON.stringify(payload);
+  }
+
+  return {
+    method: 'POST',
+    headers,
+    body: new URLSearchParams(formParams),
+    signal,
+    credentials: 'same-origin',
+  };
+};
+
+const countNewlines = (value: Uint8Array): number =>
+  value.filter(byte => byte === NEWLINE_BYTE).length;
+
+const createBlob = (
+  chunks: Uint8Array[],
+  receivedLength: number,
+  exportType: string,
+): Blob => {
+  const completeData = new Uint8Array(receivedLength);
+  let position = 0;
+  for (const chunk of chunks) {
+    completeData.set(chunk, position);
+    position += chunk.length;
+  }
+
+  const mimeType =
+    exportType === 'csv'
+      ? 'text/csv;charset=utf-8'
+      : 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
+
+  return new Blob([completeData], { type: mimeType });
+};
+
+export const useStreamingExport = (options: UseStreamingExportOptions = {}) => {
+  const [progress, setProgress] = useState<StreamingProgress>({
+    rowsProcessed: 0,
+    totalRows: undefined,
+    totalSize: 0,
+    speed: 0,
+    mbPerSecond: 0,
+    elapsedTime: 0,
+    status: ExportStatus.STREAMING,
+  });
+  const [retryCount, setRetryCount] = useState(0);
+  const abortControllerRef = useRef<AbortController | null>(null);
+  const lastExportParamsRef = useRef<StreamingExportParams | null>(null);
+  const currentBlobUrlRef = useRef<string | null>(null);
+  const isExportingRef = useRef(false);
+
+  const updateProgress = useCallback((updates: Partial<StreamingProgress>) => {
+    setProgress(prev => ({ ...prev, ...updates }));
+  }, []);
+
+  const executeExport = useCallback(
+    async (params: StreamingExportParams) => {
+      const { url, payload, filename, exportType, expectedRows } = params;
+
+      abortControllerRef.current = new AbortController();
+
+      updateProgress({
+        rowsProcessed: 0,
+        totalRows: expectedRows,
+        totalSize: 0,
+        speed: 0,
+        mbPerSecond: 0,
+        elapsedTime: 0,
+        status: ExportStatus.STREAMING,
+        filename,
+      });
+
+      try {
+        const defaultFilename = `export.${exportType}`;
+        const finalFilename = filename || defaultFilename;
+
+        const fetchOptions = await createFetchRequest(
+          url,
+          payload,
+          finalFilename,
+          exportType,
+          expectedRows,
+          abortControllerRef.current.signal,
+        );
+
+        const response = await fetch(url, fetchOptions);
+
+        if (!response.ok) {
+          throw new Error(
+            `Export failed: ${response.status} ${response.statusText}`,
+          );
+        }
+
+        if (!response.body) {
+          throw new Error('Response body is not available for streaming');
+        }
+
+        const reader = response.body.getReader();
+        const chunks: Uint8Array[] = [];
+        let receivedLength = 0;
+        let rowsProcessed = 0;
+        let hasError = false;
+
+        // eslint-disable-next-line no-constant-condition
+        while (true) {
+          // eslint-disable-next-line no-await-in-loop
+          const { done, value } = await reader.read();
+

Review Comment:
   Is there a way we could avoid disabling these ESLint rules here?Maybe we can refactor the loop logic



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,403 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/* eslint-disable theme-colors/no-literal-colors */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+  position: relative;
+`;
+
+const ProgressWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  gap: ${({ theme }) => theme.sizeUnit * 3}px;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  color: #52c41a;
+  font-size: 24px;
+  flex-shrink: 0;
+`;
+
+const ErrorIconWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 16px;
+  height: 16px;
+  background-color: #ff4d4f;
+  border-radius: 50%;
+  flex-shrink: 0;
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  color: white;
+  font-size: 10px;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)`
+  background-color: #f0fff8;
+  color: #1c997a;
+  border-color: #f0fff8;
+
+  &:hover {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+
+  &:focus {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+`;
+
+const DownloadButton = styled(Button)`
+  &.ant-btn-primary {
+    background-color: #2ec196;
+    border-color: #2ec196;
+    color: #ffffff;
+
+    &:hover:not(:disabled) {
+      background-color: #26a880;
+      border-color: #26a880;
+      color: #ffffff;
+    }
+
+    &:focus:not(:disabled) {
+      background-color: #2ec196;
+      border-color: #2ec196;
+      color: #ffffff;
+    }
+
+    &:disabled {
+      background-color: #f2f2f2;
+      border-color: #f2f2f2;
+      color: #b5b5b5;
+    }
+  }
+`;

Review Comment:
   Would it be possible to use theme color tokens here instead of hardcoded hex values?



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,403 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/* eslint-disable theme-colors/no-literal-colors */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+  position: relative;
+`;
+
+const ProgressWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  gap: ${({ theme }) => theme.sizeUnit * 3}px;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  color: #52c41a;
+  font-size: 24px;
+  flex-shrink: 0;
+`;
+
+const ErrorIconWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 16px;
+  height: 16px;
+  background-color: #ff4d4f;
+  border-radius: 50%;
+  flex-shrink: 0;
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  color: white;
+  font-size: 10px;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)`
+  background-color: #f0fff8;
+  color: #1c997a;
+  border-color: #f0fff8;
+
+  &:hover {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+
+  &:focus {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+`;
+
+const DownloadButton = styled(Button)`
+  &.ant-btn-primary {
+    background-color: #2ec196;
+    border-color: #2ec196;
+    color: #ffffff;
+
+    &:hover:not(:disabled) {
+      background-color: #26a880;
+      border-color: #26a880;
+      color: #ffffff;
+    }
+
+    &:focus:not(:disabled) {
+      background-color: #2ec196;
+      border-color: #2ec196;
+      color: #ffffff;
+    }
+
+    &:disabled {
+      background-color: #f2f2f2;
+      border-color: #f2f2f2;
+      color: #b5b5b5;
+    }
+  }
+`;
+
+interface ModalStateContentProps {
+  status: ExportStatus;
+  progress: StreamingProgress;
+  onCancel: () => void;
+  onRetry?: () => void;
+  onDownload: () => void;
+  getProgressPercentage: () => number;
+}
+
+const ErrorContent = ({
+  error,
+  onCancel,
+  onRetry,
+  getProgressPercentage,
+}: {
+  error?: string;
+  onCancel: () => void;
+  onRetry?: () => void;
+  getProgressPercentage: () => number;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <ProgressWrapper>
+        <Progress
+          percent={getProgressPercentage()}
+          status="exception"
+          showInfo={false}
+          style={{ flex: 1 }}
+        />
+        <ErrorIconWrapper>
+          <ErrorIconStyled />
+        </ErrorIconWrapper>
+      </ProgressWrapper>
+      <ErrorText type="danger">{error || t('Export failed')}</ErrorText>
+    </ProgressSection>
+    <ActionButtons>
+      <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+      {onRetry && (
+        <DownloadButton type="primary" onClick={onRetry}>
+          {t('Retry')}
+        </DownloadButton>
+      )}
+    </ActionButtons>
+  </ModalContent>
+);
+
+const CancelledContent = ({
+  getProgressPercentage,
+  onCancel,
+  onRetry,
+}: {
+  getProgressPercentage: () => number;
+  onCancel: () => void;
+  onRetry?: () => void;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <Progress
+        percent={getProgressPercentage()}
+        status="exception"
+        showInfo={false}
+      />
+      <ProgressText>{t('Export cancelled')}</ProgressText>
+    </ProgressSection>
+    <ActionButtons>
+      <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+      {onRetry && (
+        <DownloadButton type="primary" onClick={onRetry}>
+          {t('Retry')}
+        </DownloadButton>
+      )}
+    </ActionButtons>
+  </ModalContent>
+);
+
+const CompletedContent = ({
+  filename,
+  downloadUrl,
+  onCancel,
+  onDownload,
+}: {
+  filename?: string;
+  downloadUrl?: string;
+  onCancel: () => void;
+  onDownload: () => void;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <ProgressWrapper>
+        <Progress
+          percent={100}
+          status="success"
+          showInfo={false}
+          style={{ flex: 1 }}

Review Comment:
   I think here as well



##########
superset/commands/sql_lab/streaming_export_command.py:
##########
@@ -0,0 +1,239 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of SQL Lab query results."""
+
+from __future__ import annotations
+
+import csv
+import io
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+from flask_babel import gettext as __
+
+from superset import db
+from superset.commands.base import BaseCommand
+from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
+from superset.exceptions import SupersetErrorException, SupersetSecurityException
+from superset.models.sql_lab import Query
+from superset.sql.parse import SQLScript
+from superset.sqllab.limiting_factor import LimitingFactor
+
+if TYPE_CHECKING:
+    pass
+
+logger = logging.getLogger(__name__)
+
+
+class StreamingSqlResultExportCommand(BaseCommand):
+    """
+    Command to execute a streaming CSV export of SQL Lab query results.
+
+    This command handles the business logic for:
+    - Fetching SQL Lab query results
+    - Generating CSV data in chunks
+    - Managing database connections
+    - Buffering data for efficient streaming
+    """
+
+    def __init__(
+        self,
+        client_id: str,
+        chunk_size: int = 1000,
+    ):
+        """
+        Initialize the streaming export command.
+
+        Args:
+            client_id: The SQL Lab query client ID
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        self._client_id = client_id
+        self._chunk_size = chunk_size
+        self._current_app = app._get_current_object()
+        self._query: Query | None = None
+
+    def validate(self) -> None:
+        """Validate permissions and query existence."""
+        self._query = (
+            db.session.query(Query).filter_by(client_id=self._client_id).one_or_none()
+        )
+        if self._query is None:
+            raise SupersetErrorException(
+                SupersetError(
+                    message=__(
+                        "The query associated with these results could not be found. "
+                        "You need to re-run the original query."
+                    ),
+                    error_type=SupersetErrorType.RESULTS_BACKEND_ERROR,
+                    level=ErrorLevel.ERROR,
+                ),
+                status=404,
+            )
+
+        try:
+            self._query.raise_for_access()
+        except SupersetSecurityException as ex:
+            raise SupersetErrorException(
+                SupersetError(
+                    message=__("Cannot access the query"),
+                    error_type=SupersetErrorType.QUERY_SECURITY_ACCESS_ERROR,
+                    level=ErrorLevel.ERROR,
+                ),
+                status=403,
+            ) from ex
+
+    def run(self) -> Callable[[], Generator[str, None, None]]:  # noqa: C901
+        """
+        Execute the streaming CSV export.
+
+        Returns:
+            A callable that returns a generator yielding CSV data chunks as strings.
+            The callable is needed to maintain Flask app context during streaming.
+        """
+        # Load all Query attributes while session is still active
+        # to avoid DetachedInstanceError
+        assert self._query is not None
+
+        select_sql = self._query.select_sql
+        executed_sql = self._query.executed_sql
+        limiting_factor = self._query.limiting_factor
+        database = self._query.database
+
+        # Get the SQL and limit
+        if select_sql:
+            sql = select_sql
+            limit = None
+        else:
+            sql = executed_sql
+            script = SQLScript(sql, database.db_engine_spec.engine)
+            # when a query has multiple statements only the last one returns data
+            limit = script.statements[-1].get_limit_value()
+
+        if limit is not None and limiting_factor in {
+            LimitingFactor.QUERY,
+            LimitingFactor.DROPDOWN,
+            LimitingFactor.QUERY_AND_DROPDOWN,
+        }:
+            # remove extra row from `increased_limit`
+            limit -= 1
+
+        def csv_generator() -> Generator[str, None, None]:  # noqa: C901
+            """Generator that yields CSV data from SQL Lab query results."""
+            with self._current_app.app_context():
+                start_time = time.time()
+                total_bytes = 0
+
+                try:
+                    # Create a new session to keep database object attached
+                    with db.session() as session:
+                        # Merge database to prevent DetachedInstanceError
+                        merged_database = session.merge(database)
+
+                        # Execute query with streaming
+                        with merged_database.get_sqla_engine() as engine:
+                            connection = engine.connect()
+
+                            try:
+                                from sqlalchemy import text
+
+                                result_proxy = connection.execution_options(
+                                    stream_results=True
+                                ).execute(text(sql))
+
+                                columns = list(result_proxy.keys())
+
+                                # Use StringIO with csv.writer for proper escaping
+                                buffer = io.StringIO()
+                                csv_writer = csv.writer(
+                                    buffer, quoting=csv.QUOTE_MINIMAL
+                                )
+
+                                # Write CSV header
+                                csv_writer.writerow(columns)
+                                header_data = buffer.getvalue()
+                                total_bytes += len(header_data.encode("utf-8"))
+                                yield header_data
+                                buffer.seek(0)
+                                buffer.truncate()
+
+                                row_count = 0
+                                flush_threshold = 65536  # 64KB
+
+                                while True:
+                                    rows = result_proxy.fetchmany(self._chunk_size)
+                                    if not rows:
+                                        break
+
+                                    for row in rows:
+                                        # Apply limit if specified
+                                        if limit is not None and row_count >= limit:
+                                            break
+
+                                        csv_writer.writerow(row)
+                                        row_count += 1
+
+                                        # Check buffer size and flush if needed
+                                        current_size = buffer.tell()
+                                        if current_size >= flush_threshold:
+                                            data = buffer.getvalue()
+                                            data_bytes = len(data.encode("utf-8"))
+                                            total_bytes += data_bytes
+                                            yield data
+                                            buffer.seek(0)
+                                            buffer.truncate()
+
+                                    # Break outer loop if limit reached
+                                    if limit is not None and row_count >= limit:
+                                        break
+
+                                # Flush remaining buffer
+                                remaining_data = buffer.getvalue()
+                                if remaining_data:
+                                    total_bytes += len(remaining_data.encode("utf-8"))
+                                    yield remaining_data
+
+                                # Log completion
+                                total_time = time.time() - start_time
+                                total_mb = total_bytes / (1024 * 1024)
+                                logger.info(
+                                    "SQL Lab streaming CSV completed: %s rows, "
+                                    "%.1fMB in %.2fs",
+                                    f"{row_count:,}",
+                                    total_mb,
+                                    total_time,
+                                )
+
+                            finally:
+                                connection.close()
+
+                except Exception as e:
+                    logger.error("Error in SQL Lab streaming CSV generator: %s", e)
+                    import traceback
+
+                    logger.error("Traceback: %s", traceback.format_exc())
+
+                    # Send error marker for frontend to detect
+                    error_marker = (
+                        "__STREAM_ERROR__:Export failed. "
+                        "Please try again in some time.\n"

Review Comment:
   I saw Kasia say it should be "Export failed.Please try again". Should it apply here as well?



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,403 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/* eslint-disable theme-colors/no-literal-colors */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+  position: relative;
+`;
+
+const ProgressWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  gap: ${({ theme }) => theme.sizeUnit * 3}px;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  color: #52c41a;
+  font-size: 24px;
+  flex-shrink: 0;
+`;
+
+const ErrorIconWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 16px;
+  height: 16px;
+  background-color: #ff4d4f;
+  border-radius: 50%;
+  flex-shrink: 0;
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  color: white;
+  font-size: 10px;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)`
+  background-color: #f0fff8;
+  color: #1c997a;
+  border-color: #f0fff8;
+
+  &:hover {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+
+  &:focus {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+`;
+
+const DownloadButton = styled(Button)`
+  &.ant-btn-primary {
+    background-color: #2ec196;
+    border-color: #2ec196;
+    color: #ffffff;
+
+    &:hover:not(:disabled) {
+      background-color: #26a880;
+      border-color: #26a880;
+      color: #ffffff;
+    }
+
+    &:focus:not(:disabled) {
+      background-color: #2ec196;
+      border-color: #2ec196;
+      color: #ffffff;
+    }
+
+    &:disabled {
+      background-color: #f2f2f2;
+      border-color: #f2f2f2;
+      color: #b5b5b5;
+    }
+  }
+`;
+
+interface ModalStateContentProps {
+  status: ExportStatus;
+  progress: StreamingProgress;
+  onCancel: () => void;
+  onRetry?: () => void;
+  onDownload: () => void;
+  getProgressPercentage: () => number;
+}
+
+const ErrorContent = ({
+  error,
+  onCancel,
+  onRetry,
+  getProgressPercentage,
+}: {
+  error?: string;
+  onCancel: () => void;
+  onRetry?: () => void;
+  getProgressPercentage: () => number;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <ProgressWrapper>
+        <Progress
+          percent={getProgressPercentage()}
+          status="exception"
+          showInfo={false}
+          style={{ flex: 1 }}

Review Comment:
   Consider replacing inline style={{ flex: 1 }} with a styled wrapper for consistency.



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef, useEffect } from 'react';
+import { SupersetClient } from '@superset-ui/core';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportPayload {
+  [key: string]: unknown;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: StreamingExportPayload;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}
+
+const NEWLINE_BYTE = 10; // '\n' character code
+
+const createFetchRequest = async (
+  _url: string,
+  payload: StreamingExportPayload,
+  filename: string,
+  _exportType: string,
+  expectedRows: number | undefined,
+  signal: AbortSignal,
+): Promise<RequestInit> => {
+  const headers: Record<string, string> = {
+    'Content-Type': 'application/x-www-form-urlencoded',
+  };
+
+  // Get CSRF token using SupersetClient
+  const csrfToken = await SupersetClient.getCSRFToken();
+  if (csrfToken) {
+    headers['X-CSRFToken'] = csrfToken;
+  }
+
+  // Build form data - if payload has client_id, it's SQL Lab export
+  // Otherwise it's a chart export with form_data
+  const formParams: Record<string, string> = {
+    filename,
+    expected_rows: expectedRows?.toString() || '',
+  };
+
+  if ('client_id' in payload) {
+    // SQL Lab export - pass client_id directly
+    formParams.client_id = String(payload.client_id);
+  } else {
+    // Chart export - wrap payload in form_data
+    formParams.form_data = JSON.stringify(payload);
+  }
+
+  return {
+    method: 'POST',
+    headers,
+    body: new URLSearchParams(formParams),
+    signal,
+    credentials: 'same-origin',
+  };
+};
+
+const countNewlines = (value: Uint8Array): number =>
+  value.filter(byte => byte === NEWLINE_BYTE).length;
+
+const createBlob = (
+  chunks: Uint8Array[],
+  receivedLength: number,
+  exportType: string,
+): Blob => {
+  const completeData = new Uint8Array(receivedLength);
+  let position = 0;
+  for (const chunk of chunks) {
+    completeData.set(chunk, position);
+    position += chunk.length;
+  }
+
+  const mimeType =
+    exportType === 'csv'
+      ? 'text/csv;charset=utf-8'
+      : 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
+
+  return new Blob([completeData], { type: mimeType });
+};
+
+export const useStreamingExport = (options: UseStreamingExportOptions = {}) => {
+  const [progress, setProgress] = useState<StreamingProgress>({
+    rowsProcessed: 0,
+    totalRows: undefined,
+    totalSize: 0,
+    speed: 0,
+    mbPerSecond: 0,
+    elapsedTime: 0,
+    status: ExportStatus.STREAMING,
+  });
+  const [retryCount, setRetryCount] = useState(0);
+  const abortControllerRef = useRef<AbortController | null>(null);
+  const lastExportParamsRef = useRef<StreamingExportParams | null>(null);
+  const currentBlobUrlRef = useRef<string | null>(null);
+  const isExportingRef = useRef(false);
+
+  const updateProgress = useCallback((updates: Partial<StreamingProgress>) => {
+    setProgress(prev => ({ ...prev, ...updates }));
+  }, []);
+
+  const executeExport = useCallback(
+    async (params: StreamingExportParams) => {
+      const { url, payload, filename, exportType, expectedRows } = params;
+
+      abortControllerRef.current = new AbortController();
+
+      updateProgress({
+        rowsProcessed: 0,
+        totalRows: expectedRows,
+        totalSize: 0,
+        speed: 0,
+        mbPerSecond: 0,
+        elapsedTime: 0,
+        status: ExportStatus.STREAMING,
+        filename,
+      });
+
+      try {
+        const defaultFilename = `export.${exportType}`;
+        const finalFilename = filename || defaultFilename;
+
+        const fetchOptions = await createFetchRequest(
+          url,
+          payload,
+          finalFilename,
+          exportType,
+          expectedRows,
+          abortControllerRef.current.signal,
+        );
+
+        const response = await fetch(url, fetchOptions);
+
+        if (!response.ok) {
+          throw new Error(
+            `Export failed: ${response.status} ${response.statusText}`,
+          );
+        }
+
+        if (!response.body) {
+          throw new Error('Response body is not available for streaming');
+        }
+
+        const reader = response.body.getReader();
+        const chunks: Uint8Array[] = [];
+        let receivedLength = 0;
+        let rowsProcessed = 0;
+        let hasError = false;
+
+        // eslint-disable-next-line no-constant-condition
+        while (true) {
+          // eslint-disable-next-line no-await-in-loop
+          const { done, value } = await reader.read();
+
+          if (done) {
+            break;
+          }
+
+          if (abortControllerRef.current?.signal.aborted) {
+            throw new Error('Export cancelled by user');
+          }
+
+          // Check for error marker in the chunk
+          const textDecoder = new TextDecoder();
+          const chunkText = textDecoder.decode(value);
+
+          if (chunkText.includes('__STREAM_ERROR__')) {
+            const errorMatch = chunkText.match(/__STREAM_ERROR__:(.+)/);
+            const errorMsg = errorMatch
+              ? errorMatch[1].trim()
+              : 'Export failed. Please try again.';
+
+            // Update progress to show error with current progress preserved
+            updateProgress({
+              status: ExportStatus.ERROR,
+              error: errorMsg,
+              rowsProcessed,
+              totalRows: expectedRows,
+              totalSize: receivedLength,
+            });
+
+            isExportingRef.current = false;
+            options.onError?.(errorMsg);
+            hasError = true;
+            break;
+          }
+
+          chunks.push(value);
+          receivedLength += value.length;
+
+          // Count newlines using filter (more efficient than loop)
+          // Note: This counts all newlines, including those within quoted CSV fields.
+          // For an exact row count, server should send row count in response headers.
+          rowsProcessed += countNewlines(value);
+
+          // Update progress based on rows processed
+          updateProgress({
+            status: ExportStatus.STREAMING,
+            rowsProcessed,
+            totalRows: expectedRows,
+            totalSize: receivedLength,
+          });
+        }
+
+        // Check if we exited early due to error marker
+        if (hasError) {
+          return;
+        }
+
+        const blob = createBlob(chunks, receivedLength, exportType);
+
+        if (currentBlobUrlRef.current) {
+          URL.revokeObjectURL(currentBlobUrlRef.current);
+        }
+
+        const downloadUrl = URL.createObjectURL(blob);
+        currentBlobUrlRef.current = downloadUrl;
+
+        updateProgress({
+          status: ExportStatus.COMPLETED,
+          downloadUrl,
+          filename: finalFilename,
+        });
+
+        isExportingRef.current = false;
+        options.onComplete?.(downloadUrl, finalFilename);
+      } catch (error) {
+        const errorMessage =
+          error instanceof Error ? error.message : 'Unknown error occurred';
+
+        if (
+          errorMessage.includes('cancelled') ||
+          errorMessage.includes('aborted')
+        ) {
+          updateProgress({
+            status: ExportStatus.CANCELLED,
+          });
+          isExportingRef.current = false;
+        } else {
+          updateProgress({
+            status: ExportStatus.ERROR,
+            error: errorMessage,
+          });
+          options.onError?.(errorMessage);
+          isExportingRef.current = false;

Review Comment:
   This repeats itself a number of times "isExportingRef.current = false;" Maybe we could wrap it in a helper function then replace the different calls?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516494565

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19264566837) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2471972791


##########
superset-frontend/src/dashboard/components/gridComponents/Chart/Chart.jsx:
##########
@@ -157,6 +159,10 @@ const Chart = props => {
   const maxRows = useSelector(
     state => state.dashboardInfo.common.conf.SQL_MAX_ROW,
   );
+  const streamingThreshold = useSelector(
+    state =>
+      state.dashboardInfo.common.conf.CSV_STREAMING_ROW_THRESHOLD || 100000,

Review Comment:
   I liked the idea of it being configurable if someone wants to enable streaming at different thresholds as per needs , Should we make it hardcoded @geido  or is it fine as configured



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2429843961


##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app
+
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count = None

Review Comment:
   ```suggestion
           actual_row_count: int | None = None
   ```



##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app
+
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    actual_row_count = data[0].get("rowcount")
+
+        # Fallback to row_limit if actual count not available
+        if actual_row_count is None:
+            if form_data and "row_limit" in form_data:
+                actual_row_count = form_data.get("row_limit", 0)
+            elif query_context.form_data and "row_limit" in query_context.form_data:
+                actual_row_count = query_context.form_data.get("row_limit", 0)
+
+        # Use streaming if row count meets or exceeds threshold
+        if actual_row_count is not None and actual_row_count >= threshold:
+            return True
+
+        return False
+
+    def _create_streaming_csv_response(
+        self,
+        result: dict[Any, Any],
+        form_data: dict[str, Any] | None = None,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large datasets."""
+        from datetime import datetime
+
+        from flask import Response

Review Comment:
   Same here, move these to top-level.



##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app
+
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    actual_row_count = data[0].get("rowcount")
+
+        # Fallback to row_limit if actual count not available
+        if actual_row_count is None:
+            if form_data and "row_limit" in form_data:
+                actual_row_count = form_data.get("row_limit", 0)
+            elif query_context.form_data and "row_limit" in query_context.form_data:
+                actual_row_count = query_context.form_data.get("row_limit", 0)
+
+        # Use streaming if row count meets or exceeds threshold
+        if actual_row_count is not None and actual_row_count >= threshold:
+            return True
+
+        return False
+
+    def _create_streaming_csv_response(
+        self,
+        result: dict[Any, Any],
+        form_data: dict[str, Any] | None = None,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large datasets."""
+        from datetime import datetime
+
+        from flask import Response
+
+        from superset.commands.chart.data.streaming_export_command import (
+            StreamingCSVExportCommand,
+        )

Review Comment:
   And move this to top-level if possible (no circular imports).



##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app
+
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    actual_row_count = data[0].get("rowcount")
+
+        # Fallback to row_limit if actual count not available
+        if actual_row_count is None:
+            if form_data and "row_limit" in form_data:
+                actual_row_count = form_data.get("row_limit", 0)
+            elif query_context.form_data and "row_limit" in query_context.form_data:
+                actual_row_count = query_context.form_data.get("row_limit", 0)
+
+        # Use streaming if row count meets or exceeds threshold
+        if actual_row_count is not None and actual_row_count >= threshold:
+            return True
+
+        return False
+
+    def _create_streaming_csv_response(
+        self,
+        result: dict[Any, Any],
+        form_data: dict[str, Any] | None = None,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large datasets."""
+        from datetime import datetime
+
+        from flask import Response
+
+        from superset.commands.chart.data.streaming_export_command import (
+            StreamingCSVExportCommand,
+        )
+
+        query_context = result["query_context"]
+
+        # Use filename from frontend if provided, otherwise generate one
+        if not filename:
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            chart_name = "export"
+
+            if form_data and form_data.get("slice_name"):
+                chart_name = form_data["slice_name"]
+            elif form_data and form_data.get("viz_type"):
+                chart_name = form_data["viz_type"]
+
+            # Sanitize chart name for filename
+            safe_chart_name = "".join(
+                c for c in chart_name if c.isalnum() or c in ("-", "_")
+            )
+            filename = f"superset_{safe_chart_name}_{timestamp}.csv"
+
+        logger.info(
+            "Creating streaming CSV response: %s (from frontend: %s)",
+            filename,
+            filename is not None,
+        )
+        if expected_rows:
+            logger.info("Using expected_rows from frontend: %d", expected_rows)
+
+        # Execute streaming command
+        chunk_size = 1000
+        command = StreamingCSVExportCommand(query_context, chunk_size)
+        command.validate()
+
+        # Get the callable that returns the generator
+        csv_generator_callable = command.run()
+
+        # Get encoding from config
+        encoding = app.config.get("CSV_EXPORT", {}).get("encoding", "utf-8")
+
+        # Create response with streaming headers
+        response = Response(
+            csv_generator_callable(),  # Call the callable to get generator
+            mimetype=f"text/csv; charset={encoding}",
+            headers={
+                "Content-Disposition": f'attachment; filename="{filename}"',
+                "Cache-Control": "no-cache",
+                "X-Accel-Buffering": "no",  # Disable nginx buffering
+                "X-Superset-Streaming": "true",  # Identify streaming responses

Review Comment:
   Do we need this custom header? I assume Flask is setting `Transfer-Encoding: chunked` in the response, can we use that instead? (or add it, if not present)



##########
superset/commands/sql_lab/streaming_export_command.py:
##########
@@ -0,0 +1,239 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of SQL Lab query results."""
+
+from __future__ import annotations
+
+import csv
+import io
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+from flask_babel import gettext as __
+
+from superset import db
+from superset.commands.base import BaseCommand
+from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
+from superset.exceptions import SupersetErrorException, SupersetSecurityException
+from superset.models.sql_lab import Query
+from superset.sql.parse import SQLScript
+from superset.sqllab.limiting_factor import LimitingFactor
+
+if TYPE_CHECKING:
+    pass

Review Comment:
   +1



##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app
+
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    actual_row_count = data[0].get("rowcount")
+
+        # Fallback to row_limit if actual count not available
+        if actual_row_count is None:
+            if form_data and "row_limit" in form_data:
+                actual_row_count = form_data.get("row_limit", 0)
+            elif query_context.form_data and "row_limit" in query_context.form_data:
+                actual_row_count = query_context.form_data.get("row_limit", 0)
+
+        # Use streaming if row count meets or exceeds threshold
+        if actual_row_count is not None and actual_row_count >= threshold:
+            return True
+
+        return False
+
+    def _create_streaming_csv_response(
+        self,
+        result: dict[Any, Any],
+        form_data: dict[str, Any] | None = None,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large datasets."""
+        from datetime import datetime
+
+        from flask import Response
+
+        from superset.commands.chart.data.streaming_export_command import (
+            StreamingCSVExportCommand,
+        )
+
+        query_context = result["query_context"]
+
+        # Use filename from frontend if provided, otherwise generate one
+        if not filename:
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            chart_name = "export"
+
+            if form_data and form_data.get("slice_name"):
+                chart_name = form_data["slice_name"]
+            elif form_data and form_data.get("viz_type"):
+                chart_name = form_data["viz_type"]
+
+            # Sanitize chart name for filename
+            safe_chart_name = "".join(
+                c for c in chart_name if c.isalnum() or c in ("-", "_")
+            )
+            filename = f"superset_{safe_chart_name}_{timestamp}.csv"

Review Comment:
   It's better to use `secure_filename` from `werkzeug`.



##########
superset/commands/chart/data/streaming_export_command.py:
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of large datasets."""
+
+from __future__ import annotations
+
+import csv
+import io
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+
+from superset.commands.base import BaseCommand
+
+if TYPE_CHECKING:
+    from superset.common.query_context import QueryContext
+
+logger = logging.getLogger(__name__)
+
+
+class StreamingCSVExportCommand(BaseCommand):
+    """
+    Command to execute a streaming CSV export.
+
+    This command handles the business logic for:
+    - Executing database queries with server-side cursors
+    - Generating CSV data in chunks
+    - Managing database connections
+    - Buffering data for efficient streaming
+    """
+
+    def __init__(
+        self,
+        query_context: QueryContext,
+        chunk_size: int = 1000,
+    ):
+        """
+        Initialize the streaming export command.
+
+        Args:
+            query_context: The query context containing datasource and query details
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        self._query_context = query_context
+        self._chunk_size = chunk_size
+        self._current_app = app._get_current_object()
+
+    def validate(self) -> None:
+        """Validate permissions and query context."""
+        self._query_context.raise_for_access()
+
+    def run(self) -> Callable[[], Generator[str, None, None]]:  # noqa: C901
+        """
+        Execute the streaming CSV export.
+
+        Returns:
+            A callable that returns a generator yielding CSV data chunks as strings.
+            The callable is needed to maintain Flask app context during streaming.
+        """
+
+        def csv_generator() -> Generator[str, None, None]:  # noqa: C901
+            """Generator that yields CSV data from database query."""
+            with self._current_app.app_context():
+                start_time = time.time()
+                total_bytes = 0
+
+                try:
+                    from superset import db
+                    from superset.connectors.sqla.models import SqlaTable
+
+                    datasource = self._query_context.datasource
+
+                    with db.session() as session:
+                        if isinstance(datasource, SqlaTable):
+                            datasource = session.merge(datasource)
+
+                        query_obj = self._query_context.queries[0]
+                        sql_query = datasource.get_query_str(query_obj.to_dict())
+
+                        with datasource.database.get_sqla_engine() as engine:
+                            connection = engine.connect()
+
+                            try:
+                                from sqlalchemy import text

Review Comment:
   This is a lot of nesting; it would be better to refactor this method and split it into multiple methods to avoid so many nesting levels.
   
   Also, this import can be on the top-level.



##########
superset/commands/chart/data/streaming_export_command.py:
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of large datasets."""
+
+from __future__ import annotations
+
+import csv
+import io
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+
+from superset.commands.base import BaseCommand
+
+if TYPE_CHECKING:
+    from superset.common.query_context import QueryContext
+
+logger = logging.getLogger(__name__)
+
+
+class StreamingCSVExportCommand(BaseCommand):
+    """
+    Command to execute a streaming CSV export.
+
+    This command handles the business logic for:
+    - Executing database queries with server-side cursors
+    - Generating CSV data in chunks
+    - Managing database connections
+    - Buffering data for efficient streaming
+    """
+
+    def __init__(
+        self,
+        query_context: QueryContext,
+        chunk_size: int = 1000,
+    ):
+        """
+        Initialize the streaming export command.
+
+        Args:
+            query_context: The query context containing datasource and query details
+            chunk_size: Number of rows to fetch per database query (default: 1000)
+        """
+        self._query_context = query_context
+        self._chunk_size = chunk_size
+        self._current_app = app._get_current_object()
+
+    def validate(self) -> None:
+        """Validate permissions and query context."""
+        self._query_context.raise_for_access()
+
+    def run(self) -> Callable[[], Generator[str, None, None]]:  # noqa: C901
+        """
+        Execute the streaming CSV export.
+
+        Returns:
+            A callable that returns a generator yielding CSV data chunks as strings.
+            The callable is needed to maintain Flask app context during streaming.
+        """
+
+        def csv_generator() -> Generator[str, None, None]:  # noqa: C901
+            """Generator that yields CSV data from database query."""
+            with self._current_app.app_context():
+                start_time = time.time()
+                total_bytes = 0
+
+                try:
+                    from superset import db
+                    from superset.connectors.sqla.models import SqlaTable
+
+                    datasource = self._query_context.datasource
+
+                    with db.session() as session:
+                        if isinstance(datasource, SqlaTable):
+                            datasource = session.merge(datasource)
+
+                        query_obj = self._query_context.queries[0]
+                        sql_query = datasource.get_query_str(query_obj.to_dict())
+
+                        with datasource.database.get_sqla_engine() as engine:
+                            connection = engine.connect()
+
+                            try:
+                                from sqlalchemy import text
+
+                                result_proxy = connection.execution_options(
+                                    stream_results=True
+                                ).execute(text(sql_query))
+
+                                columns = list(result_proxy.keys())
+
+                                # Use StringIO with csv.writer for proper escaping
+                                buffer = io.StringIO()
+                                csv_writer = csv.writer(
+                                    buffer, quoting=csv.QUOTE_MINIMAL
+                                )
+
+                                # Write CSV header
+                                csv_writer.writerow(columns)
+                                header_data = buffer.getvalue()
+                                total_bytes += len(header_data.encode("utf-8"))
+                                yield header_data
+                                buffer.seek(0)
+                                buffer.truncate()
+
+                                row_count = 0
+                                flush_threshold = 65536  # 64KB
+
+                                while True:
+                                    rows = result_proxy.fetchmany(self._chunk_size)
+                                    if not rows:
+                                        break

Review Comment:
   You can simply do:
   
   ```python
   while rows := result_proxy.fetchmany(self._chunk_size):
       ...
   ```



##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app

Review Comment:
   You should be able to have this at the top of the file.



##########
superset/sqllab/api.py:
##########
@@ -294,6 +294,130 @@ def export_csv(self, client_id: str) -> CsvResponse:
         )
         return response
 
+    @expose("/export_streaming/", methods=("POST",))
+    @protect()
+    @permission_name("read")
+    @statsd_metrics
+    @event_logger.log_this_with_context(
+        action=lambda self,
+        *args,
+        **kwargs: f"{self.__class__.__name__}.export_streaming_csv",
+        log_to_statsd=False,
+    )
+    def export_streaming_csv(self) -> Response:
+        """Export SQL query results using streaming for large datasets.
+        ---
+        post:
+          summary: Export SQL query results to CSV with streaming
+          requestBody:
+            description: Export parameters
+            required: true
+            content:
+              application/x-www-form-urlencoded:
+                schema:
+                  type: object
+                  properties:
+                    client_id:
+                      type: string
+                      description: The SQL query result identifier
+                    filename:
+                      type: string
+                      description: Optional filename for the export
+                    expected_rows:
+                      type: integer
+                      description: Optional expected row count for progress tracking
+          responses:
+            200:
+              description: Streaming CSV export
+              content:
+                text/csv:
+                  schema:
+                    type: string
+            400:
+              $ref: '#/components/responses/400'
+            401:
+              $ref: '#/components/responses/401'
+            403:
+              $ref: '#/components/responses/403'
+            404:
+              $ref: '#/components/responses/404'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        # Extract parameters from form data
+        client_id = request.form.get("client_id")
+        filename = request.form.get("filename")
+
+        if not client_id:
+            return self.response_400(message="client_id is required")
+
+        expected_rows = None
+        if expected_rows_str := request.form.get("expected_rows"):
+            try:
+                expected_rows = int(expected_rows_str)
+            except (ValueError, TypeError):
+                logger.warning("Invalid expected_rows value: %s", expected_rows_str)
+
+        return self._create_streaming_csv_response(client_id, filename, expected_rows)
+
+    def _create_streaming_csv_response(
+        self,
+        client_id: str,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large SQL Lab result sets."""
+        from datetime import datetime
+
+        from superset.commands.sql_lab.streaming_export_command import (
+            StreamingSqlResultExportCommand,
+        )
+
+        # Execute streaming command
+        chunk_size = 1000
+        command = StreamingSqlResultExportCommand(client_id, chunk_size)
+        command.validate()
+
+        # Generate filename if not provided
+        if not filename:
+            query = command._query
+            assert query is not None
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            safe_name = "".join(
+                c for c in (query.name or "query") if c.isalnum() or c in ("-", "_")
+            )
+            filename = f"sqllab_{safe_name}_{timestamp}.csv"
+
+        # Get the callable that returns the generator
+        csv_generator_callable = command.run()
+
+        # Get encoding from config
+        encoding = app.config.get("CSV_EXPORT", {}).get("encoding", "utf-8")
+
+        # Create response with streaming headers
+        response = Response(
+            csv_generator_callable(),  # Call the callable to get generator
+            mimetype=f"text/csv; charset={encoding}",
+            headers={
+                "Content-Disposition": f'attachment; filename="{filename}"',
+                "Cache-Control": "no-cache",
+                "X-Accel-Buffering": "no",  # Disable nginx buffering
+                "X-Superset-Streaming": "true",  # Identify streaming responses

Review Comment:
   Same question here about using a standard header.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2429509638


##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef, useEffect } from 'react';
+import { SupersetClient } from '@superset-ui/core';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportPayload {
+  [key: string]: unknown;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: StreamingExportPayload;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}
+
+const NEWLINE_BYTE = 10; // '\n' character code
+
+const createFetchRequest = async (
+  _url: string,
+  payload: StreamingExportPayload,
+  filename: string,
+  _exportType: string,
+  expectedRows: number | undefined,
+  signal: AbortSignal,
+): Promise<RequestInit> => {
+  const headers: Record<string, string> = {
+    'Content-Type': 'application/x-www-form-urlencoded',
+  };
+
+  // Get CSRF token using SupersetClient
+  const csrfToken = await SupersetClient.getCSRFToken();
+  if (csrfToken) {
+    headers['X-CSRFToken'] = csrfToken;
+  }
+
+  // Build form data - if payload has client_id, it's SQL Lab export
+  // Otherwise it's a chart export with form_data
+  const formParams: Record<string, string> = {
+    filename,
+    expected_rows: expectedRows?.toString() || '',
+  };
+
+  if ('client_id' in payload) {
+    // SQL Lab export - pass client_id directly
+    formParams.client_id = String(payload.client_id);
+  } else {
+    // Chart export - wrap payload in form_data
+    formParams.form_data = JSON.stringify(payload);
+  }
+
+  return {
+    method: 'POST',
+    headers,
+    body: new URLSearchParams(formParams),
+    signal,
+    credentials: 'same-origin',
+  };
+};
+
+const countNewlines = (value: Uint8Array): number =>
+  value.filter(byte => byte === NEWLINE_BYTE).length;
+
+const createBlob = (
+  chunks: Uint8Array[],
+  receivedLength: number,
+  exportType: string,
+): Blob => {
+  const completeData = new Uint8Array(receivedLength);
+  let position = 0;
+  for (const chunk of chunks) {
+    completeData.set(chunk, position);
+    position += chunk.length;
+  }
+
+  const mimeType =
+    exportType === 'csv'
+      ? 'text/csv;charset=utf-8'
+      : 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
+
+  return new Blob([completeData], { type: mimeType });
+};
+
+export const useStreamingExport = (options: UseStreamingExportOptions = {}) => {
+  const [progress, setProgress] = useState<StreamingProgress>({
+    rowsProcessed: 0,
+    totalRows: undefined,
+    totalSize: 0,
+    speed: 0,
+    mbPerSecond: 0,
+    elapsedTime: 0,
+    status: ExportStatus.STREAMING,
+  });
+  const [retryCount, setRetryCount] = useState(0);
+  const abortControllerRef = useRef<AbortController | null>(null);
+  const lastExportParamsRef = useRef<StreamingExportParams | null>(null);
+  const currentBlobUrlRef = useRef<string | null>(null);
+  const isExportingRef = useRef(false);
+
+  const updateProgress = useCallback((updates: Partial<StreamingProgress>) => {
+    setProgress(prev => ({ ...prev, ...updates }));
+  }, []);
+
+  const executeExport = useCallback(
+    async (params: StreamingExportParams) => {
+      const { url, payload, filename, exportType, expectedRows } = params;
+
+      abortControllerRef.current = new AbortController();
+
+      updateProgress({
+        rowsProcessed: 0,
+        totalRows: expectedRows,
+        totalSize: 0,
+        speed: 0,
+        mbPerSecond: 0,
+        elapsedTime: 0,
+        status: ExportStatus.STREAMING,
+        filename,
+      });
+
+      try {
+        const defaultFilename = `export.${exportType}`;
+        const finalFilename = filename || defaultFilename;
+
+        const fetchOptions = await createFetchRequest(
+          url,
+          payload,
+          finalFilename,
+          exportType,
+          expectedRows,
+          abortControllerRef.current.signal,
+        );
+
+        const response = await fetch(url, fetchOptions);
+
+        if (!response.ok) {
+          throw new Error(
+            `Export failed: ${response.status} ${response.statusText}`,
+          );
+        }
+
+        if (!response.body) {
+          throw new Error('Response body is not available for streaming');
+        }
+
+        const reader = response.body.getReader();
+        const chunks: Uint8Array[] = [];
+        let receivedLength = 0;
+        let rowsProcessed = 0;
+        let hasError = false;
+
+        // eslint-disable-next-line no-constant-condition
+        while (true) {
+          // eslint-disable-next-line no-await-in-loop
+          const { done, value } = await reader.read();
+

Review Comment:
   I think this makes sense here, since the reads can't be done in parallel. If that's the case [it should be fine to disable the rule](https://eslint.org/docs/latest/rules/no-await-in-loop#when-not-to-use-it).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516806644

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19266247505) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516806632

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19266247487) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516820233

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/19266247505) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)
   
   β€’ **Environment:** http://34.220.88.207:8080 (admin/admin)
   β€’ **Lifetime:** 48h auto-cleanup
   β€’ **Updates:** New commits create fresh environments automatically


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516806614

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19266247526) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3527388023

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/19329778277) for [ed5c1e3](https://github.com/apache/superset/commit/ed5c1e3)
   
   β€’ **Environment:** http://35.92.165.125:8080 (admin/admin)
   β€’ **Lifetime:** 48h auto-cleanup
   β€’ **Updates:** New commits create fresh environments automatically


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2469133730


##########
superset-frontend/src/dashboard/components/gridComponents/Chart/Chart.jsx:
##########
@@ -157,6 +159,10 @@ const Chart = props => {
   const maxRows = useSelector(
     state => state.dashboardInfo.common.conf.SQL_MAX_ROW,
   );
+  const streamingThreshold = useSelector(
+    state =>
+      state.dashboardInfo.common.conf.CSV_STREAMING_ROW_THRESHOLD || 100000,

Review Comment:
   Should this be a constant default or a config even?



##########
superset-frontend/src/SqlLab/components/ResultSet/index.tsx:
##########
@@ -351,21 +395,47 @@ const ResultSet = ({
                 css={copyButtonStyles}
                 buttonSize="small"
                 buttonStyle="secondary"
-                href={getExportCsvUrl(query.id)}
+                href={
+                  !shouldUseStreamingExport()
+                    ? getExportCsvUrl(query.id)
+                    : undefined
+                }
                 data-test="export-csv-button"
-                onClick={() => {
-                  logAction(LOG_ACTIONS_SQLLAB_DOWNLOAD_CSV, {});
-                  if (
-                    limitingFactor === LimitingFactor.Dropdown &&
-                    limit === rowsCount
-                  ) {
-                    Modal.warning({
-                      title: t('Download is on the way'),
-                      content: t(
-                        'Downloading %(rows)s rows based on the LIMIT configuration. If you want the entire result set, you need to adjust the LIMIT.',
-                        { rows: rowsCount.toLocaleString() },
-                      ),
+                onClick={e => {
+                  const useStreaming = shouldUseStreamingExport();
+
+                  if (useStreaming) {
+                    e.preventDefault();
+                    setShowStreamingModal(true);
+
+                    const timestamp = new Date()
+                      .toISOString()
+                      .slice(0, 19)
+                      .replace(/[-:]/g, '')
+                      .replace('T', '_');
+                    const filename = `sqllab_${query.id}_${timestamp}.csv`;

Review Comment:
   Should this be handled on the backend?



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,405 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { styled, t, useTheme } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';

Review Comment:
   Can we import from the core instead of antd directly?



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,403 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/* eslint-disable theme-colors/no-literal-colors */
+import { styled, t } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+  position: relative;
+`;
+
+const ProgressWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  gap: ${({ theme }) => theme.sizeUnit * 3}px;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  color: #52c41a;
+  font-size: 24px;
+  flex-shrink: 0;
+`;
+
+const ErrorIconWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 16px;
+  height: 16px;
+  background-color: #ff4d4f;
+  border-radius: 50%;
+  flex-shrink: 0;
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  color: white;
+  font-size: 10px;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)`
+  background-color: #f0fff8;
+  color: #1c997a;
+  border-color: #f0fff8;
+
+  &:hover {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+
+  &:focus {
+    background-color: #f0fff8;
+    color: #1c997a;
+    border-color: #1c997a;
+  }
+`;
+
+const DownloadButton = styled(Button)`
+  &.ant-btn-primary {
+    background-color: #2ec196;
+    border-color: #2ec196;
+    color: #ffffff;
+
+    &:hover:not(:disabled) {
+      background-color: #26a880;
+      border-color: #26a880;
+      color: #ffffff;
+    }
+
+    &:focus:not(:disabled) {
+      background-color: #2ec196;
+      border-color: #2ec196;
+      color: #ffffff;
+    }
+
+    &:disabled {
+      background-color: #f2f2f2;
+      border-color: #f2f2f2;
+      color: #b5b5b5;
+    }
+  }
+`;
+
+interface ModalStateContentProps {
+  status: ExportStatus;
+  progress: StreamingProgress;
+  onCancel: () => void;
+  onRetry?: () => void;
+  onDownload: () => void;
+  getProgressPercentage: () => number;
+}
+
+const ErrorContent = ({
+  error,
+  onCancel,
+  onRetry,
+  getProgressPercentage,
+}: {
+  error?: string;
+  onCancel: () => void;
+  onRetry?: () => void;
+  getProgressPercentage: () => number;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <ProgressWrapper>
+        <Progress
+          percent={getProgressPercentage()}
+          status="exception"
+          showInfo={false}
+          style={{ flex: 1 }}

Review Comment:
   I don't think this is resolved



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,405 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { styled, t, useTheme } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+  position: relative;
+`;
+
+const ProgressWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  gap: ${({ theme }) => theme.sizeUnit * 3}px;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  color: ${({ theme }) => theme.colorSuccess};
+  font-size: 24px;
+  flex-shrink: 0;
+`;
+
+const ErrorIconWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 16px;
+  height: 16px;
+  background-color: ${({ theme }) => theme.colorError};
+  border-radius: 50%;
+  flex-shrink: 0;
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  color: ${({ theme }) => theme.colorWhite};
+  font-size: 10px;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)`
+  background-color: ${({ theme }) => theme.colorSuccessBg};
+  color: ${({ theme }) => theme.colorSuccess};
+  border-color: ${({ theme }) => theme.colorSuccessBg};
+
+  &:hover {
+    background-color: ${({ theme }) => theme.colorSuccessBg};
+    color: ${({ theme }) => theme.colorSuccess};
+    border-color: ${({ theme }) => theme.colorSuccess};
+  }
+
+  &:focus {
+    background-color: ${({ theme }) => theme.colorSuccessBg};
+    color: ${({ theme }) => theme.colorSuccess};
+    border-color: ${({ theme }) => theme.colorSuccess};
+  }
+`;
+
+const DownloadButton = styled(Button)`
+  &.ant-btn-primary {
+    background-color: ${({ theme }) => theme.colorSuccess};
+    border-color: ${({ theme }) => theme.colorSuccess};
+    color: ${({ theme }) => theme.colorWhite};
+
+    &:hover:not(:disabled) {
+      background-color: ${({ theme }) => theme.colorSuccessActive};
+      border-color: ${({ theme }) => theme.colorSuccessActive};
+      color: ${({ theme }) => theme.colorWhite};
+    }
+
+    &:focus:not(:disabled) {
+      background-color: ${({ theme }) => theme.colorSuccess};
+      border-color: ${({ theme }) => theme.colorSuccess};
+      color: ${({ theme }) => theme.colorWhite};
+    }
+
+    &:disabled {
+      background-color: ${({ theme }) => theme.colorBgContainerDisabled};
+      border-color: ${({ theme }) => theme.colorBgContainerDisabled};
+      color: ${({ theme }) => theme.colorTextDisabled};
+    }
+  }
+`;
+
+interface ModalStateContentProps {
+  status: ExportStatus;
+  progress: StreamingProgress;
+  onCancel: () => void;
+  onRetry?: () => void;
+  onDownload: () => void;
+  getProgressPercentage: () => number;
+}
+
+const ErrorContent = ({
+  error,
+  onCancel,
+  onRetry,
+  getProgressPercentage,
+}: {
+  error?: string;
+  onCancel: () => void;
+  onRetry?: () => void;
+  getProgressPercentage: () => number;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <ProgressWrapper>
+        <Progress
+          percent={getProgressPercentage()}
+          status="exception"
+          showInfo={false}
+          style={{ flex: 1 }}
+        />
+        <ErrorIconWrapper>
+          <ErrorIconStyled />
+        </ErrorIconWrapper>
+      </ProgressWrapper>
+      <ErrorText type="danger">{error || t('Export failed')}</ErrorText>
+    </ProgressSection>
+    <ActionButtons>
+      <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+      {onRetry && (
+        <DownloadButton type="primary" onClick={onRetry}>
+          {t('Retry')}
+        </DownloadButton>
+      )}
+    </ActionButtons>
+  </ModalContent>
+);
+
+const CancelledContent = ({
+  getProgressPercentage,
+  onCancel,
+  onRetry,
+}: {
+  getProgressPercentage: () => number;
+  onCancel: () => void;
+  onRetry?: () => void;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <Progress
+        percent={getProgressPercentage()}
+        status="exception"
+        showInfo={false}
+      />
+      <ProgressText>{t('Export cancelled')}</ProgressText>
+    </ProgressSection>
+    <ActionButtons>
+      <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+      {onRetry && (
+        <DownloadButton type="primary" onClick={onRetry}>
+          {t('Retry')}
+        </DownloadButton>
+      )}
+    </ActionButtons>
+  </ModalContent>
+);
+
+const CompletedContent = ({
+  filename,
+  downloadUrl,
+  onCancel,
+  onDownload,
+}: {
+  filename?: string;
+  downloadUrl?: string;
+  onCancel: () => void;
+  onDownload: () => void;
+}) => (
+  <ModalContent>
+    <ProgressSection>
+      <ProgressWrapper>
+        <Progress
+          percent={100}
+          status="success"
+          showInfo={false}
+          style={{ flex: 1 }}
+        />
+        <SuccessIcon />
+      </ProgressWrapper>
+      <ProgressText>
+        {t('Export successful: %s', filename || 'export')}
+      </ProgressText>
+    </ProgressSection>
+    <ActionButtons>
+      <CancelButton onClick={onCancel}>{t('Close')}</CancelButton>
+      <DownloadButton
+        type="primary"
+        onClick={onDownload}
+        disabled={!downloadUrl}
+      >
+        {t('Download')}
+      </DownloadButton>
+    </ActionButtons>
+  </ModalContent>
+);
+
+const StreamingContent = ({
+  filename,
+  getProgressPercentage,
+  onCancel,
+}: {
+  filename?: string;
+  getProgressPercentage: () => number;
+  onCancel: () => void;
+}) => {
+  const theme = useTheme();
+  return (
+    <ModalContent>
+      <ProgressSection>
+        <Progress
+          percent={getProgressPercentage()}
+          status="normal"
+          strokeColor={theme.colorSuccess}
+          showInfo
+          format={percent => `${Math.round(percent || 0)}%`}
+        />
+        <ProgressText>
+          {filename
+            ? t('Processing export for %s', filename)
+            : t('Processing export...')}
+        </ProgressText>
+      </ProgressSection>
+      <ActionButtons>
+        <CancelButton onClick={onCancel}>{t('Cancel')}</CancelButton>
+        <DownloadButton type="primary" disabled>
+          {t('Download')}
+        </DownloadButton>
+      </ActionButtons>
+    </ModalContent>
+  );
+};
+
+const ModalStateContent = ({
+  status,
+  progress,
+  onCancel,
+  onRetry,
+  onDownload,
+  getProgressPercentage,
+}: ModalStateContentProps) => {
+  const { downloadUrl, filename, error } = progress;
+
+  switch (status) {
+    case ExportStatus.ERROR:
+      return (
+        <ErrorContent
+          error={error}
+          onCancel={onCancel}
+          onRetry={onRetry}
+          getProgressPercentage={getProgressPercentage}
+        />
+      );
+    case ExportStatus.CANCELLED:
+      return (
+        <CancelledContent
+          getProgressPercentage={getProgressPercentage}
+          onCancel={onCancel}
+          onRetry={onRetry}
+        />
+      );
+    case ExportStatus.COMPLETED:
+      return (
+        <CompletedContent
+          filename={filename}
+          downloadUrl={downloadUrl}
+          onCancel={onCancel}
+          onDownload={onDownload}
+        />
+      );
+    default:
+      return (
+        <StreamingContent
+          filename={filename}
+          getProgressPercentage={getProgressPercentage}
+          onCancel={onCancel}
+        />
+      );
+  }

Review Comment:
   These components look very similar. Should we make the props conditional instead of the components? 



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,367 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef, useEffect } from 'react';
+import { SupersetClient } from '@superset-ui/core';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportPayload {
+  [key: string]: unknown;
+}

Review Comment:
   I would prefer `Record<string, unknown>` here



##########
superset-frontend/src/components/StreamingExportModal/StreamingExportModal.tsx:
##########
@@ -0,0 +1,405 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { styled, t, useTheme } from '@superset-ui/core';
+import { Modal, Button, Typography, Progress } from 'antd';
+import { Icons } from '@superset-ui/core/components/Icons';
+
+const { Text } = Typography;
+
+export enum ExportStatus {
+  STREAMING = 'streaming',
+  COMPLETED = 'completed',
+  ERROR = 'error',
+  CANCELLED = 'cancelled',
+}
+
+export interface StreamingProgress {
+  totalRows?: number;
+  rowsProcessed: number;
+  totalSize: number;
+  status: ExportStatus;
+  downloadUrl?: string;
+  error?: string;
+  filename?: string;
+  speed?: number;
+  mbPerSecond?: number;
+  elapsedTime?: number;
+  retryCount?: number;
+}
+
+interface StreamingExportModalProps {
+  visible: boolean;
+  onCancel: () => void;
+  onRetry?: () => void;
+  progress: StreamingProgress;
+}
+
+const ModalContent = styled.div`
+  padding: ${({ theme }) => theme.sizeUnit * 4}px 0
+    ${({ theme }) => theme.sizeUnit * 2}px;
+`;
+
+const ProgressSection = styled.div`
+  margin: ${({ theme }) => theme.sizeUnit * 6}px 0;
+  position: relative;
+`;
+
+const ProgressWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  gap: ${({ theme }) => theme.sizeUnit * 3}px;
+`;
+
+const SuccessIcon = styled(Icons.CheckCircleFilled)`
+  color: ${({ theme }) => theme.colorSuccess};
+  font-size: 24px;
+  flex-shrink: 0;
+`;
+
+const ErrorIconWrapper = styled.div`
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 16px;
+  height: 16px;
+  background-color: ${({ theme }) => theme.colorError};
+  border-radius: 50%;
+  flex-shrink: 0;
+`;
+
+const ErrorIconStyled = styled(Icons.CloseOutlined)`
+  color: ${({ theme }) => theme.colorWhite};
+  font-size: 10px;
+`;
+
+const ActionButtons = styled.div`
+  display: flex;
+  gap: ${({ theme }) => theme.sizeUnit * 2}px;
+  justify-content: flex-end;
+`;
+
+const ProgressText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const ErrorText = styled(Text)`
+  display: block;
+  text-align: center;
+  margin-top: ${({ theme }) => theme.sizeUnit * 4}px;
+`;
+
+const CancelButton = styled(Button)`
+  background-color: ${({ theme }) => theme.colorSuccessBg};
+  color: ${({ theme }) => theme.colorSuccess};
+  border-color: ${({ theme }) => theme.colorSuccessBg};
+
+  &:hover {
+    background-color: ${({ theme }) => theme.colorSuccessBg};
+    color: ${({ theme }) => theme.colorSuccess};
+    border-color: ${({ theme }) => theme.colorSuccess};
+  }
+
+  &:focus {
+    background-color: ${({ theme }) => theme.colorSuccessBg};
+    color: ${({ theme }) => theme.colorSuccess};
+    border-color: ${({ theme }) => theme.colorSuccess};
+  }
+`;
+
+const DownloadButton = styled(Button)`
+  &.ant-btn-primary {
+    background-color: ${({ theme }) => theme.colorSuccess};
+    border-color: ${({ theme }) => theme.colorSuccess};
+    color: ${({ theme }) => theme.colorWhite};
+
+    &:hover:not(:disabled) {
+      background-color: ${({ theme }) => theme.colorSuccessActive};
+      border-color: ${({ theme }) => theme.colorSuccessActive};
+      color: ${({ theme }) => theme.colorWhite};
+    }
+
+    &:focus:not(:disabled) {
+      background-color: ${({ theme }) => theme.colorSuccess};
+      border-color: ${({ theme }) => theme.colorSuccess};
+      color: ${({ theme }) => theme.colorWhite};
+    }
+
+    &:disabled {
+      background-color: ${({ theme }) => theme.colorBgContainerDisabled};
+      border-color: ${({ theme }) => theme.colorBgContainerDisabled};
+      color: ${({ theme }) => theme.colorTextDisabled};
+    }
+  }

Review Comment:
   Can we clean this up a bit? use a single function and maybe check if some of this is even necessary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2472120296


##########
superset-frontend/src/dashboard/components/gridComponents/Chart/Chart.jsx:
##########
@@ -157,6 +159,10 @@ const Chart = props => {
   const maxRows = useSelector(
     state => state.dashboardInfo.common.conf.SQL_MAX_ROW,
   );
+  const streamingThreshold = useSelector(
+    state =>
+      state.dashboardInfo.common.conf.CSV_STREAMING_ROW_THRESHOLD || 100000,

Review Comment:
   Oh, i like it configured. I was talking about the 100000 default



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516494583

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19264566485) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] (feat-streaming): Streaming Csv Uploads for over 100k records [superset]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3364807959

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/35478?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `2.43056%` with `281 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 46.59%. Comparing base ([`96170e4`](https://app.codecov.io/gh/apache/superset/commit/96170e43c0330890ed50a617e91249e870a13973?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b78895a`](https://app.codecov.io/gh/apache/superset/commit/b78895a30ddaa1c0d9cd982ecbe5939f4c5adf88?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/35478?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [superset/views/streaming.py](https://app.codecov.io/gh/apache/superset/pull/35478?src=pr&el=tree&filepath=superset%2Fviews%2Fstreaming.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmlld3Mvc3RyZWFtaW5nLnB5) | 0.00% | [149 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/35478?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [superset/views/streaming\_progress.py](https://app.codecov.io/gh/apache/superset/pull/35478?src=pr&el=tree&filepath=superset%2Fviews%2Fstreaming_progress.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmlld3Mvc3RyZWFtaW5nX3Byb2dyZXNzLnB5) | 0.00% | [95 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/35478?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [superset/charts/data/api.py](https://app.codecov.io/gh/apache/superset/pull/35478?src=pr&el=tree&filepath=superset%2Fcharts%2Fdata%2Fapi.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY2hhcnRzL2RhdGEvYXBpLnB5) | 15.90% | [37 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/35478?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #35478       +/-   ##
   ===========================================
   + Coverage        0   46.59%   +46.59%     
   ===========================================
     Files           0      591      +591     
     Lines           0    43851    +43851     
     Branches        0     4742     +4742     
   ===========================================
   + Hits            0    20431    +20431     
   - Misses          0    23092    +23092     
   - Partials        0      328      +328     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/superset/pull/35478/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ” | |
   |---|---|---|
   | [hive](https://app.codecov.io/gh/apache/superset/pull/35478/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.01% <2.43%> (?)` | |
   | [python](https://app.codecov.io/gh/apache/superset/pull/35478/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.59% <2.43%> (?)` | |
   | [unit](https://app.codecov.io/gh/apache/superset/pull/35478/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ΓΈ> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/35478?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3386434144

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/18380946650) for [f9a72c8](https://github.com/apache/superset/commit/f9a72c8)
   
   β€’ **Environment:** http://34.221.177.73:8080 (admin/admin)
   β€’ **Lifetime:** 48h auto-cleanup
   β€’ **Updates:** New commits create fresh environments automatically


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3511637467

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/19232767048) for [3fba394](https://github.com/apache/superset/commit/3fba394)
   
   β€’ **Environment:** http://52.27.203.26:8080 (admin/admin)
   β€’ **Lifetime:** 48h auto-cleanup
   β€’ **Updates:** New commits create fresh environments automatically


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3527338482

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19329778277) for [ed5c1e3](https://github.com/apache/superset/commit/ed5c1e3)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] (feat-streaming): Streaming Csv Uploads for over 100k records [superset]

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2405054205


##########
superset/charts/data/api.py:
##########
@@ -462,3 +485,108 @@
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app
+
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    actual_row_count = data[0].get("rowcount")
+
+        # Fallback to row_limit if actual count not available
+        if actual_row_count is None:
+            if form_data and "row_limit" in form_data:
+                actual_row_count = form_data.get("row_limit", 0)
+            elif query_context.form_data and "row_limit" in query_context.form_data:
+                actual_row_count = query_context.form_data.get("row_limit", 0)
+
+        # Use streaming if row count meets or exceeds threshold
+        if actual_row_count is not None and actual_row_count >= threshold:
+            return True
+
+        return False
+
+    def _create_streaming_csv_response(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None, filename: str | None = None, expected_rows: int | None = None
+    ) -> Response:
+        """Create a streaming CSV response for large datasets."""
+        from datetime import datetime
+
+        from flask import Response
+
+        from superset.commands.chart.data.streaming_export_command import (
+            StreamingCSVExportCommand,
+        )
+
+        query_context = result["query_context"]
+
+        # Use filename from frontend if provided, otherwise generate one
+        if not filename:
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            chart_name = "export"
+
+            if form_data and form_data.get("slice_name"):
+                chart_name = form_data["slice_name"]
+            elif form_data and form_data.get("viz_type"):
+                chart_name = form_data["viz_type"]
+
+            # Sanitize chart name for filename
+            safe_chart_name = "".join(
+                c for c in chart_name if c.isalnum() or c in ("-", "_")
+            )
+            filename = f"superset_{safe_chart_name}_{timestamp}.csv"
+
+        logger.info("Creating streaming CSV response: %s (from frontend: %s)", filename, filename is not None)
+        if expected_rows:
+            logger.info("πŸ“Š Using expected_rows from frontend: %d", expected_rows)
+
+        # Execute streaming command
+        chunk_size = 1000
+        command = StreamingCSVExportCommand(query_context, chunk_size)
+        command.validate()
+
+        # Get the callable that returns the generator
+        csv_generator_callable = command.run()
+
+        # Get encoding from config
+        encoding = app.config.get("CSV_EXPORT", {}).get("encoding", "utf-8")
+
+        # Create response with streaming headers
+        response = Response(
+            csv_generator_callable(),  # Call the callable to get generator

Review Comment:
   ## Information exposure through an exception
   
   [Stack trace information](1) flows to this location and may be exposed to an external user.
   
   [Show more details](https://github.com/apache/superset/security/code-scanning/2061)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] (feat-streaming): Streaming Csv Uploads for over 100k records [superset]

Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3364785197

   Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment `/korbit-review`.
   
   > Your admin can change your review schedule in the [Korbit Console](https://app.korbit.ai/settings)
   
   
   
   [](5c3ba35d-1f50-40ee-9820-7f03026d6f38)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2472424059


##########
superset-frontend/src/dashboard/components/gridComponents/Chart/Chart.jsx:
##########
@@ -157,6 +159,10 @@ const Chart = props => {
   const maxRows = useSelector(
     state => state.dashboardInfo.common.conf.SQL_MAX_ROW,
   );
+  const streamingThreshold = useSelector(
+    state =>
+      state.dashboardInfo.common.conf.CSV_STREAMING_ROW_THRESHOLD || 100000,

Review Comment:
   Ohh ok , yes nice catch , will move it constant



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "LevisNgigi (via GitHub)" <gi...@apache.org>.
LevisNgigi commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2430477439


##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef, useEffect } from 'react';
+import { SupersetClient } from '@superset-ui/core';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportPayload {
+  [key: string]: unknown;
+}
+
+interface StreamingExportParams {
+  url: string;
+  payload: StreamingExportPayload;
+  filename?: string;
+  exportType: 'csv' | 'xlsx';
+  expectedRows?: number;
+}
+
+const NEWLINE_BYTE = 10; // '\n' character code
+
+const createFetchRequest = async (
+  _url: string,
+  payload: StreamingExportPayload,
+  filename: string,
+  _exportType: string,
+  expectedRows: number | undefined,
+  signal: AbortSignal,
+): Promise<RequestInit> => {
+  const headers: Record<string, string> = {
+    'Content-Type': 'application/x-www-form-urlencoded',
+  };
+
+  // Get CSRF token using SupersetClient
+  const csrfToken = await SupersetClient.getCSRFToken();
+  if (csrfToken) {
+    headers['X-CSRFToken'] = csrfToken;
+  }
+
+  // Build form data - if payload has client_id, it's SQL Lab export
+  // Otherwise it's a chart export with form_data
+  const formParams: Record<string, string> = {
+    filename,
+    expected_rows: expectedRows?.toString() || '',
+  };
+
+  if ('client_id' in payload) {
+    // SQL Lab export - pass client_id directly
+    formParams.client_id = String(payload.client_id);
+  } else {
+    // Chart export - wrap payload in form_data
+    formParams.form_data = JSON.stringify(payload);
+  }
+
+  return {
+    method: 'POST',
+    headers,
+    body: new URLSearchParams(formParams),
+    signal,
+    credentials: 'same-origin',
+  };
+};
+
+const countNewlines = (value: Uint8Array): number =>
+  value.filter(byte => byte === NEWLINE_BYTE).length;
+
+const createBlob = (
+  chunks: Uint8Array[],
+  receivedLength: number,
+  exportType: string,
+): Blob => {
+  const completeData = new Uint8Array(receivedLength);
+  let position = 0;
+  for (const chunk of chunks) {
+    completeData.set(chunk, position);
+    position += chunk.length;
+  }
+
+  const mimeType =
+    exportType === 'csv'
+      ? 'text/csv;charset=utf-8'
+      : 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
+
+  return new Blob([completeData], { type: mimeType });
+};
+
+export const useStreamingExport = (options: UseStreamingExportOptions = {}) => {
+  const [progress, setProgress] = useState<StreamingProgress>({
+    rowsProcessed: 0,
+    totalRows: undefined,
+    totalSize: 0,
+    speed: 0,
+    mbPerSecond: 0,
+    elapsedTime: 0,
+    status: ExportStatus.STREAMING,
+  });
+  const [retryCount, setRetryCount] = useState(0);
+  const abortControllerRef = useRef<AbortController | null>(null);
+  const lastExportParamsRef = useRef<StreamingExportParams | null>(null);
+  const currentBlobUrlRef = useRef<string | null>(null);
+  const isExportingRef = useRef(false);
+
+  const updateProgress = useCallback((updates: Partial<StreamingProgress>) => {
+    setProgress(prev => ({ ...prev, ...updates }));
+  }, []);
+
+  const executeExport = useCallback(
+    async (params: StreamingExportParams) => {
+      const { url, payload, filename, exportType, expectedRows } = params;
+
+      abortControllerRef.current = new AbortController();
+
+      updateProgress({
+        rowsProcessed: 0,
+        totalRows: expectedRows,
+        totalSize: 0,
+        speed: 0,
+        mbPerSecond: 0,
+        elapsedTime: 0,
+        status: ExportStatus.STREAMING,
+        filename,
+      });
+
+      try {
+        const defaultFilename = `export.${exportType}`;
+        const finalFilename = filename || defaultFilename;
+
+        const fetchOptions = await createFetchRequest(
+          url,
+          payload,
+          finalFilename,
+          exportType,
+          expectedRows,
+          abortControllerRef.current.signal,
+        );
+
+        const response = await fetch(url, fetchOptions);
+
+        if (!response.ok) {
+          throw new Error(
+            `Export failed: ${response.status} ${response.statusText}`,
+          );
+        }
+
+        if (!response.body) {
+          throw new Error('Response body is not available for streaming');
+        }
+
+        const reader = response.body.getReader();
+        const chunks: Uint8Array[] = [];
+        let receivedLength = 0;
+        let rowsProcessed = 0;
+        let hasError = false;
+
+        // eslint-disable-next-line no-constant-condition
+        while (true) {
+          // eslint-disable-next-line no-await-in-loop
+          const { done, value } = await reader.read();
+

Review Comment:
   Thank you for the clarification



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3503505781

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19174467104) for [3fba394](https://github.com/apache/superset/commit/3fba394)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516806635

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19266247481) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3516806629

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/19266247451) for [b5393a4](https://github.com/apache/superset/commit/b5393a4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "amaannawab923 (via GitHub)" <gi...@apache.org>.
amaannawab923 closed pull request #35478: feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage
URL: https://github.com/apache/superset/pull/35478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35478:
URL: https://github.com/apache/superset/pull/35478#issuecomment-3383324292

   πŸŽͺ [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/18358486742) for [c6d1838](https://github.com/apache/superset/commit/c6d1838)
   
   β€’ **Environment:** http://34.222.113.141:8080 (admin/admin)
   β€’ **Lifetime:** 48h auto-cleanup
   β€’ **Updates:** New commits create fresh environments automatically


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2412835130


##########
superset/sqllab/api.py:
##########
@@ -294,6 +294,128 @@
         )
         return response
 
+    @expose("/export_streaming/", methods=("POST",))
+    @statsd_metrics
+    @event_logger.log_this_with_context(
+        action=lambda self,
+        *args,
+        **kwargs: f"{self.__class__.__name__}.export_streaming_csv",
+        log_to_statsd=False,
+    )
+    def export_streaming_csv(self) -> Response:
+        """Export SQL query results using streaming for large datasets.
+        ---
+        post:
+          summary: Export SQL query results to CSV with streaming
+          requestBody:
+            description: Export parameters
+            required: true
+            content:
+              application/x-www-form-urlencoded:
+                schema:
+                  type: object
+                  properties:
+                    client_id:
+                      type: string
+                      description: The SQL query result identifier
+                    filename:
+                      type: string
+                      description: Optional filename for the export
+                    expected_rows:
+                      type: integer
+                      description: Optional expected row count for progress tracking
+          responses:
+            200:
+              description: Streaming CSV export
+              content:
+                text/csv:
+                  schema:
+                    type: string
+            400:
+              $ref: '#/components/responses/400'
+            401:
+              $ref: '#/components/responses/401'
+            403:
+              $ref: '#/components/responses/403'
+            404:
+              $ref: '#/components/responses/404'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        # Extract parameters from form data
+        client_id = request.form.get("client_id")
+        filename = request.form.get("filename")
+
+        if not client_id:
+            return self.response_400(message="client_id is required")
+
+        expected_rows = None
+        if expected_rows_str := request.form.get("expected_rows"):
+            try:
+                expected_rows = int(expected_rows_str)
+            except (ValueError, TypeError):
+                logger.warning("Invalid expected_rows value: %s", expected_rows_str)
+
+        return self._create_streaming_csv_response(client_id, filename, expected_rows)
+
+    def _create_streaming_csv_response(
+        self,
+        client_id: str,
+        filename: str | None = None,
+        expected_rows: int | None = None,
+    ) -> Response:
+        """Create a streaming CSV response for large SQL Lab result sets."""
+        from datetime import datetime
+
+        from superset.commands.sql_lab.streaming_export_command import (
+            StreamingSqlResultExportCommand,
+        )
+
+        # Execute streaming command
+        chunk_size = 1000
+        command = StreamingSqlResultExportCommand(client_id, chunk_size)
+        command.validate()
+
+        # Generate filename if not provided
+        if not filename:
+            query = command._query
+            assert query is not None
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            safe_name = "".join(
+                c for c in (query.name or "query") if c.isalnum() or c in ("-", "_")
+            )
+            filename = f"sqllab_{safe_name}_{timestamp}.csv"
+
+        # Get the callable that returns the generator
+        csv_generator_callable = command.run()
+
+        # Get encoding from config
+        encoding = app.config.get("CSV_EXPORT", {}).get("encoding", "utf-8")
+
+        # Create response with streaming headers
+        response = Response(
+            csv_generator_callable(),  # Call the callable to get generator

Review Comment:
   ## Information exposure through an exception
   
   [Stack trace information](1) flows to this location and may be exposed to an external user.
   
   [Show more details](https://github.com/apache/superset/security/code-scanning/2062)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage [superset]

Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2428789159


##########
superset-frontend/src/SqlLab/components/ResultSet/index.tsx:
##########
@@ -300,6 +318,32 @@ const ResultSet = ({
   const getExportCsvUrl = (clientId: string) =>
     `/api/v1/sqllab/export/${clientId}/`;
 
+  const getStreamingExportUrl = () => `/api/v1/sqllab/export_streaming/`;
+

Review Comment:
   I don t think there s a need to save the url like this, for `getExportCsvUrl` made sense because it changes depending on the clientId



##########
superset-frontend/src/SqlLab/components/ResultSet/index.tsx:
##########
@@ -222,11 +228,23 @@ const ResultSet = ({
   const [searchText, setSearchText] = useState('');
   const [cachedData, setCachedData] = useState<Record<string, unknown>[]>([]);
   const [showSaveDatasetModal, setShowSaveDatasetModal] = useState(false);
+  const [showStreamingModal, setShowStreamingModal] = useState(false);
 
   const history = useHistory();
   const dispatch = useDispatch();
   const logAction = useLogAction({ queryId, sqlEditorId: query.sqlEditorId });
 
+  // Streaming export hook

Review Comment:
   ```suggestion
     
   ```



##########
superset-frontend/src/SqlLab/components/ResultSet/index.tsx:
##########
@@ -300,6 +318,32 @@ const ResultSet = ({
   const getExportCsvUrl = (clientId: string) =>
     `/api/v1/sqllab/export/${clientId}/`;
 
+  const getStreamingExportUrl = () => `/api/v1/sqllab/export_streaming/`;
+
+  const handleCloseStreamingModal = () => {
+    setShowStreamingModal(false);
+    resetExport();
+  };
+
+  // Check if streaming export should be used
+  const shouldUseStreamingExport = () => {

Review Comment:
   No need for these comments



##########
superset/charts/data/api.py:
##########
@@ -425,7 +441,25 @@ def _get_data_response(
         except ChartDataQueryFailedError as exc:
             return self.response_400(message=exc.message)
 
-        return self._send_chart_response(result, form_data, datasource)
+        return self._send_chart_response(
+            result, form_data, datasource, filename, expected_rows
+        )
+
+    def _extract_export_params_from_request(self) -> tuple[str | None, int | None]:
+        """Extract filename and expected_rows from request for streaming exports."""
+        filename = request.form.get("filename")
+        if filename:
+            logger.info("πŸ“ FRONTEND PROVIDED FILENAME: %s", filename)
+
+        expected_rows = None
+        if expected_rows_str := request.form.get("expected_rows"):
+            try:
+                expected_rows = int(expected_rows_str)
+                logger.info("πŸ“Š FRONTEND PROVIDED EXPECTED ROWS: %d", expected_rows)

Review Comment:
   We don t use emojis in loggers



##########
superset-frontend/src/dashboard/components/gridComponents/Chart/Chart.jsx:
##########
@@ -544,6 +616,21 @@ const Chart = props => {
           emitCrossFilters={emitCrossFilters}
         />
       </ChartWrapper>
+
+      {/* Streaming Export Modal */}

Review Comment:
   No need for this comment



##########
superset-frontend/src/SqlLab/components/ResultSet/index.tsx:
##########
@@ -351,21 +395,47 @@ const ResultSet = ({
                 css={copyButtonStyles}
                 buttonSize="small"
                 buttonStyle="secondary"
-                href={getExportCsvUrl(query.id)}
+                href={
+                  !shouldUseStreamingExport()
+                    ? getExportCsvUrl(query.id)
+                    : undefined
+                }

Review Comment:
   Can we try something like => href={!shouldUseStreamingExport() && getExportCsvUrl(query.id)}



##########
superset-frontend/src/SqlLab/components/ResultSet/index.tsx:
##########
@@ -351,21 +395,47 @@ const ResultSet = ({
                 css={copyButtonStyles}
                 buttonSize="small"
                 buttonStyle="secondary"
-                href={getExportCsvUrl(query.id)}
+                href={
+                  !shouldUseStreamingExport()
+                    ? getExportCsvUrl(query.id)
+                    : undefined
+                }
                 data-test="export-csv-button"
-                onClick={() => {
-                  logAction(LOG_ACTIONS_SQLLAB_DOWNLOAD_CSV, {});
-                  if (
-                    limitingFactor === LimitingFactor.Dropdown &&
-                    limit === rowsCount
-                  ) {
-                    Modal.warning({
-                      title: t('Download is on the way'),
-                      content: t(
-                        'Downloading %(rows)s rows based on the LIMIT configuration. If you want the entire result set, you need to adjust the LIMIT.',
-                        { rows: rowsCount.toLocaleString() },
-                      ),
+                onClick={e => {
+                  const useStreaming = shouldUseStreamingExport();
+
+                  if (useStreaming) {
+                    e.preventDefault();
+                    setShowStreamingModal(true);
+
+                    const timestamp = new Date()
+                      .toISOString()
+                      .slice(0, 19)
+                      .replace(/[-:]/g, '')
+                      .replace('T', '_');
+                    const filename = `sqllab_${query.id}_${timestamp}.csv`;

Review Comment:
   Can we check if there s a util that already handles the creation of a timestamp?



##########
superset/commands/sql_lab/streaming_export_command.py:
##########
@@ -0,0 +1,239 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Command for streaming CSV exports of SQL Lab query results."""
+
+from __future__ import annotations
+
+import csv
+import io
+import logging
+import time
+from typing import Callable, Generator, TYPE_CHECKING
+
+from flask import current_app as app
+from flask_babel import gettext as __
+
+from superset import db
+from superset.commands.base import BaseCommand
+from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
+from superset.exceptions import SupersetErrorException, SupersetSecurityException
+from superset.models.sql_lab import Query
+from superset.sql.parse import SQLScript
+from superset.sqllab.limiting_factor import LimitingFactor
+
+if TYPE_CHECKING:
+    pass

Review Comment:
   I noticed that we have two streaming_export_command.py, is there a way to dry this up?



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.test.ts:
##########
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { renderHook, act } from '@testing-library/react-hooks';
+import { useStreamingExport } from './useStreamingExport';
+import { ExportStatus } from './StreamingExportModal';
+
+// Mock SupersetClient
+jest.mock('@superset-ui/core', () => ({
+  ...jest.requireActual('@superset-ui/core'),
+  SupersetClient: {
+    getCSRFToken: jest.fn(() => Promise.resolve('mock-csrf-token')),
+  },
+}));
+
+// Mock URL APIs

Review Comment:
   No need for comments here eiter



##########
superset-frontend/src/SqlLab/components/ResultSet/index.tsx:
##########
@@ -222,11 +228,23 @@ const ResultSet = ({
   const [searchText, setSearchText] = useState('');
   const [cachedData, setCachedData] = useState<Record<string, unknown>[]>([]);
   const [showSaveDatasetModal, setShowSaveDatasetModal] = useState(false);
+  const [showStreamingModal, setShowStreamingModal] = useState(false);
 
   const history = useHistory();
   const dispatch = useDispatch();
   const logAction = useLogAction({ queryId, sqlEditorId: query.sqlEditorId });
 
+  // Streaming export hook
+  const { progress, startExport, resetExport, retryExport } =
+    useStreamingExport({
+      onComplete: () => {
+        // Modal will show download button

Review Comment:
   ```suggestion
   ```



##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None
+    ) -> bool:
+        """Determine if streaming should be used based on actual row count threshold."""
+        from flask import current_app as app
+
+        query_context = result["query_context"]
+        result_format = query_context.result_format
+
+        # Only support CSV streaming currently
+        if result_format.lower() != "csv":
+            return False
+
+        # Get streaming threshold from config
+        threshold = app.config.get("CSV_STREAMING_ROW_THRESHOLD", 100000)
+
+        # Extract actual row count (same logic as frontend)
+        actual_row_count = None
+        viz_type = form_data.get("viz_type") if form_data else None
+
+        # For table viz, try to get actual row count from query results
+        if viz_type == "table" and result.get("queries"):
+            # Check if we have rowcount in the second query result (like frontend does)
+            queries = result.get("queries", [])
+            if len(queries) > 1 and queries[1].get("data"):
+                data = queries[1]["data"]
+                if isinstance(data, list) and len(data) > 0:
+                    actual_row_count = data[0].get("rowcount")
+
+        # Fallback to row_limit if actual count not available

Review Comment:
   Let s get rid of the comments everywhere, and keep only those that are needed



##########
superset/charts/data/api.py:
##########
@@ -462,3 +496,116 @@ def _create_query_context_from_form(
             return ChartDataQueryContextSchema().load(form_data)
         except KeyError as ex:
             raise ValidationError("Request is incorrect") from ex
+
+    def _should_use_streaming(
+        self, result: dict[Any, Any], form_data: dict[str, Any] | None = None

Review Comment:
   is there something better than dict[Any, Any]?



##########
superset-frontend/src/components/StreamingExportModal/useStreamingExport.ts:
##########
@@ -0,0 +1,367 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { useState, useCallback, useRef, useEffect } from 'react';
+import { SupersetClient } from '@superset-ui/core';
+import { ExportStatus, StreamingProgress } from './StreamingExportModal';
+
+interface UseStreamingExportOptions {
+  onComplete?: (downloadUrl: string, filename: string) => void;
+  onError?: (error: string) => void;
+}
+
+interface StreamingExportPayload {
+  [key: string]: unknown;
+}

Review Comment:
   Is there a better type other than unknown?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] (feat-streaming): Streaming Csv Uploads for over 100k records [superset]

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #35478:
URL: https://github.com/apache/superset/pull/35478#discussion_r2402361620


##########
superset/views/streaming.py:
##########
@@ -0,0 +1,449 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Streaming HTTP responses for large dataset exports."""
+
+from __future__ import annotations
+
+import logging
+import time
+import uuid
+from datetime import datetime
+from typing import Any, Generator, TYPE_CHECKING
+
+from flask import current_app as app, Response
+from werkzeug.datastructures import Headers
+
+from superset.views.streaming_progress import progress_tracker
+
+if TYPE_CHECKING:
+    from superset.common.query_context import QueryContext
+
+logger = logging.getLogger(__name__)
+
+
+def create_streaming_csv_response_simple(
+    data_generator: Generator[str, None, None],
+    filename: str = "export.csv",
+    encoding: str = "utf-8",
+) -> Response:
+    """
+    Create a simple streaming CSV response using Flask's standard pattern.
+
+    This follows the official Flask streaming documentation pattern.
+    """
+    from flask import Response
+
+    # Create response with proper headers
+    response = Response(
+        data_generator,

Review Comment:
   ## Information exposure through an exception
   
   [Stack trace information](1) flows to this location and may be exposed to an external user.
   
   [Show more details](https://github.com/apache/superset/security/code-scanning/2059)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org