[I] arrow-csv: no way to tell that `with_truncated_rows` padded a row [arrow-rs]

["AndreaBozzo (via GitHub)" <gi...@apache.org>]

AndreaBozzo opened a new issue, #10577:
URL: https://github.com/apache/arrow-rs/issues/10577

   ### Is your feature request related to a problem or challenge?
   
   `ReaderBuilder::with_truncated_rows(true)` repairs a short row by padding it,
   but there is no way for a caller to learn that it happened. For anything that
   reports on data quality, a repaired parse and a clean parse are different
   outcomes, and right now they are indistinguishable.
   
   The information is not merely unexposed — it is destroyed. Padding fills the
   offsets to produce **zero-length fields**
   ([`records.rs`, the `self.current_field < self.num_columns` branch](https://github.com/apache/arrow-rs/blob/main/arrow-csv/src/reader/records.rs)),
   and the null regex later turns those into nulls. By the time a `RecordBatch`
   exists, a padded field is byte-identical to a genuinely empty trailing field:
   
   ```
   name,age,city
   Alice,25,Rome  ->  ["Alice", "25", "Rome"]
   Carol,35,      ->  ["Carol", "35", NULL]   <- three fields, last one empty
   Bob,30         ->  ["Bob",   "30", NULL]   <- two fields, padded
   ```
   
   Full repro (arrow-csv 59.1.0, output above is real):
   
   ```rust
   use arrow_array::{Array, StringArray};
   use arrow_csv::ReaderBuilder;
   use arrow_schema::{DataType, Field, Schema};
   use std::io::Cursor;
   use std::sync::Arc;
   
   fn main() {
       let schema = Arc::new(Schema::new(vec![
           Field::new("name", DataType::Utf8, true),
           Field::new("age", DataType::Utf8, true),
           Field::new("city", DataType::Utf8, true),
       ]));
   
       let csv = "name,age,city\nAlice,25,Rome\nCarol,35,\nBob,30\n";
       let reader = ReaderBuilder::new(schema)
           .with_header(true)
           .with_truncated_rows(true)
           .build(Cursor::new(csv))
           .unwrap();
   
       for batch in reader {
           let batch = batch.unwrap();
           for row in 0..batch.num_rows() {
               let cells: Vec<String> = (0..batch.num_columns())
                   .map(|c| {
                       let a = batch.column(c).as_any().downcast_ref::<StringArray>().unwrap();
                       if a.is_null(row) { "NULL".into() } else { format!("{:?}", a.value(row)) }
                   })
                   .collect();
               println!("row {row}: [{}]", cells.join(", "));
           }
       }
       // Nothing here, or anywhere on the reader, reports that row 2 was padded.
   }
   ```
   
   ### Describe the solution you'd like
   
   `RecordDecoder` already knows exactly when it pads — it is the branch quoted
   above. A counter incremented there, plus an accessor on `Decoder` / `Reader` /
   `BufReader`, would be enough:
   
   ```rust
   /// The number of rows padded because they had fewer fields than the schema.
   ///
   /// Always 0 unless `with_truncated_rows(true)` was set.
   pub fn truncated_row_count(&self) -> usize
   ```
   
   Purely additive, and it costs nothing when the feature is off.
   
   ### Describe alternatives you've considered
   
   - **Inferring it from nulls in the batch.** Not possible, per the repro: a
     padded field and an empty trailing field are the same null.
   - **Disabling the null regex** (as in #6874). Doesn't help — with the regex off
     both cases become `""` instead of `NULL`, still identical.
   - **Pre-scanning the file with the `csv` crate.** What we ended up doing. It
     works and is cheaper than expected (~3% of profiling wall time on a 218 MB
     file, since the analysis dominates the parse), but it is a second full read
     of the input purely to recover a number the decoder already had.
   
   ### Additional context
   
   We hit this in [dataprof](https://github.com/AndreaBozzo/dataprof), a data
   profiler whose contract is that a repaired scan must never be reported as a
   clean one. Our Arrow-backed CSV engine was the only path that could report a
   short-row file with a perfect consistency score
   ([#470](https://github.com/AndreaBozzo/dataprof/issues/470)).
   
   Related but separate: `with_truncated_rows` is asymmetric — it recovers rows
   with *too few* fields, but a row with *too many* is always an error, with no
   equivalent opt-in to drop the surplus. Handling both directions currently means
   pre-scanning for the widest record and padding the schema out to it, then
   projecting the extra columns away. Happy to open that separately if it is worth
   its own issue.
   
   I'm willing to put up a PR for the counter if the approach sounds right.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] arrow-csv: no way to tell that `with_truncated_rows` padded a row [arrow-rs]

["Jefffrey (via GitHub)" <gi...@apache.org>]

Jefffrey closed issue #10577: arrow-csv: no way to tell that `with_truncated_rows` padded a row
URL: https://github.com/apache/arrow-rs/issues/10577


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] arrow-csv: no way to tell that `with_truncated_rows` padded a row [arrow-rs]

["Jefffrey (via GitHub)" <gi...@apache.org>]

Jefffrey commented on issue #10577:
URL: https://github.com/apache/arrow-rs/issues/10577#issuecomment-5215707290

   sounds reasonable, if youre just after the count
   
   > Related but separate: `with_truncated_rows` is asymmetric — it recovers rows
   with too few fields, but a row with too many is always an error, with no
   equivalent opt-in to drop the surplus. Handling both directions currently means
   pre-scanning for the widest record and padding the schema out to it, then
   projecting the extra columns away. Happy to open that separately if it is worth
   its own issue.
   
   this could be worth raising as a separate issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org