[I] CSV/JSONL scan sources silently drop rows that fail type parsing [texera]

["eugenegujing (via GitHub)" <gi...@apache.org>]

eugenegujing opened a new issue, #6279:
URL: https://github.com/apache/texera/issues/6279

   ### What happened?
   
   The CSV and JSONL scan source operators infer each column's type from only the first `INFER_READ_LIMIT` (= 100) rows. When a **later** row holds a value that does not parse as the inferred type, the operator **silently discards the entire row** — no warning, no skipped-row count, no console message, no log entry. The user has no way to learn that data was lost, so every downstream count, aggregate, join, and model is computed on a silently truncated dataset.
   
   Cells that are simply **empty** are kept as `null` (the row survives), while cells that **fail to parse** cause the **whole row** to vanish. The operator treats "missing" and "malformed" differently, and the more destructive path is the silent one. Rejecting malformed input is defensible but doing it **silently** would be a bug
   
   **What I expected to happen:** the loss should not be silent. At minimum the number of skipped rows should be surfaced (execution statistic / console warning), or the scan should fail loudly and name the offending row and column. Losing data with zero signal is the defect.
   
   ### How to reproduce?
   
   Using the `diabetes` dataset (400 data rows). Type inference only samples the first 100 rows, so inject a bad value **after** row 100 into a column that is all-integer in the first 100 rows (e.g. `age`), which makes inference stably INTEGER:
   
   1. Create a corrupted copy that puts a non-integer into `age` on three rows past row 100:
      ```bash
      # data rows 150, 250, 350 -> age becomes "55.5" (fails Integer parsing)
      awk -F, 'BEGIN{OFS=","} NR==1{print; next}
        {n++; if(n==150||n==250||n==350){$8="55.5"} print}' diabetes.csv > diabetes_b2.csv
      ```
   2. Upload `diabetes_b2.csv` as a dataset and point a **CSV File Scan** operator at it.
   3. Run the workflow and look at the scan operator's output row count.
   
   **Expected:** 403 rows accounted for, or a visible signal that 3 rows were skipped.
   
   **Actual:** the scan emits **400** rows. The 3 rows containing `55.5` are gone, with no warning, no skipped-row count, and nothing in the console — silent data loss. (JSONL, ParallelCSV, and the `csvOld` variant share the same behavior.)
   
   <img width="1344" height="836" alt="Image" src="https://github.com/user-attachments/assets/e8fb0c15-130a-4e2f-8069-b4c7fd2d05b0" />
   
   ### Version/Branch
   
   1.3.0-incubating-SNAPSHOT (main)
   
   ### Commit Hash (Optional)
   
   _No response_
   
   ### What browsers are you seeing the problem on?
   
   _No response_
   
   ### Relevant log output
   
   ```shell
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] CSV/JSONL scan sources silently drop rows that fail type parsing [texera]

["eugenegujing (via GitHub)" <gi...@apache.org>]

eugenegujing commented on issue #6279:
URL: https://github.com/apache/texera/issues/6279#issuecomment-4920095675

   /take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] CSV/JSONL scan sources silently drop rows that fail type parsing [texera]

["Yicong-Huang (via GitHub)" <gi...@apache.org>]

Yicong-Huang closed issue #6279: CSV/JSONL scan sources silently drop rows that fail type parsing
URL: https://github.com/apache/texera/issues/6279


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org