[I] [Bug] List reader fails with "Inconsistent sample sizes" on nullable outer rows [mahout]

["0lai0 (via GitHub)" <gi...@apache.org>]

0lai0 opened a new issue, #1401:
URL: https://github.com/apache/mahout/issues/1401

   ## What
   
   When reading a nullable `List<T>` column, a `null` **outer** row makes the entire read fail with an `InvalidInput` consistency error, instead of being treated as a missing sample.
   
   The List sample-size validation calls Arrow's `ListArray::value_length(i)` without an `is_null(i)` guard. For a `null` outer row, `value_length(i)` returns `0`; the reader treats this as a length-0 list, which trips the `sample_size` consistency check. After a non-null row it fails with `expected N, got 0`; if the `null` row is row `0`, it seeds `sample_size` to `0` and breaks every subsequent row. `FillZero` (`NullHandling::FillZero`) does not help — the failure is at the *list-length* level, before any value-filling runs.
   
   Reproduce by reading a `List<Float64>` column with the values `[[1, 2], null, [3, 4]]`:
   
   ```
   InvalidInput("Inconsistent sample sizes: expected 2, got 0")
   ```
   
   ## Why
   
   - **Real-world impact.** Nullable `List` columns with null rows are common in practice, so this is hit by ordinary data, not just edge cases.
   - **No coverage.** There is currently no test for a null outer row.
   - **Pre-existing, but worth fixing.** This is not introduced by #1393 — the previous per-row code behaved the same way. The PR is fine to merge; this is a follow-up.
   - **Affects multiple readers.** Three readers share the same gap (see How), so the bug surfaces across both batch and streaming paths and the IPC path.
   
   ## How
   
   The fix is to add an `is_null(i)` guard to the List sample-size validation so a `null` row is not counted as a length-0 list (including the row-0 seed). Affected code, all in `qdp/qdp-core/src/readers/`:
   
   - **`ParquetReader`** — `parquet.rs:294–306`. Loop over all rows calls `value_length(i)` at line 295 with no `is_null(i)` check; a null row 0 locks `sample_size` to `0`.
   - **`ParquetStreamingReader`** — `parquet.rs`. Row-0 seed at line 564 (no null check) and validation loop at lines 568–573, same gap.
   - **`ArrowIPCReader`** — `arrow_ipc.rs:154`. Uses `list_array.value(i).len()` with no `is_null(i)` guard; same problem.
   
   The intended semantics of a null row (skip vs. fill per null policy) should be decided first, since it determines how the guard behaves.
   
   ### Tasks
   
   - [ ] Add the `is_null(i)` guard to all three readers (`ParquetReader`, `ParquetStreamingReader`, `ArrowIPCReader`).
   - [ ] Add a regression test for a nullable `List` column with a `null` outer row, covering both null-in-the-middle and null-at-row-0.
   - [ ] (Optional) Harden `written / self.sample_size.unwrap_or(1)` at `parquet.rs:490` against `Some(0)`: `unwrap_or(1)` guards `None` but not `Some(0)`. Unreachable today (the consistency check errors first), but a cheap guard keeps the division safe against a future `sample_size == 0` path.
   
   ---
   
   Follow-up to #1393.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] List reader fails with "Inconsistent sample sizes" on nullable outer rows [mahout]

["guan404ming (via GitHub)" <gi...@apache.org>]

guan404ming closed issue #1401: [Bug] List<T> reader fails with "Inconsistent sample sizes" on nullable outer rows
URL: https://github.com/apache/mahout/issues/1401


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org