You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by GitBox <gi...@apache.org> on 2026/07/27 22:06:02 UTC
[GH] (texera/feat/scan-report-skipped-rows): Workflow run "Required Checks" failed!
The GitHub Actions job "Required Checks" on texera.git/feat/scan-report-skipped-rows has failed.
Run started by GitHub user eugenegujing (triggered by eugenegujing).
Head commit for run:
3ba7130a3022a70aae84f9630596d15eab300fa5 / eugenegujing <eu...@outlook.com>
feat(workflow-operator): skip and report scan rows that do not match the inferred schema
CSV/JSONL scan sources infer each column's type from only the first INFER_READ_LIMIT (=100) rows. When a later row held a value that did not parse as the inferred type, the execs silently dropped the entire row — no error, no warning, no count. Downstream counts, aggregates, and joins then ran on a silently truncated dataset.
Per the consensus in discussion #6324 (skip + surface, instead of the fail-fast approach originally proposed in #6323), a scan now still skips the unparsable row — the run completes and existing workflows keep working — but each skipped row is surfaced to the user as a console warning naming the row number, offending value, column, and expected type, e.g.:
WARNING: skipped row 150 — value '55.5' in column 'age' cannot be read as INTEGER. Column types were inferred from the first 100 rows of the file, and this value does not match.
Changes:
- Add ScanRowParseError, which builds the per-row warning by re-parsing the failing row's fields to identify the offending column, with a generic fallback when no single column is identifiable (e.g. a malformed JSON line).
- Add SkippedRowReporter, shared by all scan variants: reports the first 100 skipped rows individually, then a single summary line with the true total, so a heavily malformed file cannot flood the console or exhaust memory.
- Record-and-skip in CSVScanSourceOpExec, JSONLScanSourceOpExec, ParallelCSVScanSourceOpExec, and CSVOldScanSourceOpExec. Legitimate null paths (empty cell parsed to null, blank/all-null line, exhausted block) are untouched and not reported. Row numbers: CSV uses its existing counter, JSONL/csvOld gain absolute line/data-row numbers, ParallelCSV reports none (byte-partitioned across workers).
- Add OperatorExecutor.getWarnings (defaults to empty), a generic non-fatal warning channel for executors.
- Emit the warnings at FinalizeExecutor in DataProcessor as PRINT console messages via the new ErrorUtils.mkPrintConsoleMessage. Titles keep the "WARNING: " prefix the UI keys on. Unlike the executor exception path, this does not pause the run.
Tests: scan exec specs cover skip-and-report (row/value/column/type), the 100-detail cap with summary, empty-cell and blank-line non-reporting, and the malformed-JSON fallback; DataProcessorSpec covers the PRINT emission at finalize and asserts the run is not paused.
Closes #6279
Report URL: https://github.com/apache/texera/actions/runs/30308491583
With regards,
GitHub Actions via GitBox