[I] arrow-csv: RecordBatchReader::schema ignores column projection [arrow-rs]

["yangbaechu (via GitHub)" <gi...@apache.org>]

yangbaechu opened a new issue, #10382:
URL: https://github.com/apache/arrow-rs/issues/10382

   ### Describe the bug
   
   When an `arrow_csv::Reader` is configured with `ReaderBuilder::with_projection`, `RecordBatchReader::schema()` returns the original unprojected schema, while the yielded `RecordBatch` uses the projected schema.
   
   Therefore, the schema advertised by the reader does not match the schema of its batches.
   
   ### To Reproduce
   
   ```rust
   use std::io::Cursor;
   use std::sync::Arc;
   
   use arrow_array::RecordBatchReader;
   use arrow_csv::ReaderBuilder;
   use arrow_schema::{DataType, Field, Schema};
   
   #[test]
   fn projected_reader_schema_matches_batch_schema() {
       let schema = Arc::new(Schema::new(vec![
           Field::new("a", DataType::Int32, false),
           Field::new("b", DataType::Int32, false),
       ]));
   
       let mut reader = ReaderBuilder::new(schema)
           .with_projection(vec![1])
           .build(Cursor::new(b"1,2\n"))
           .unwrap();
   
       let reader_schema = RecordBatchReader::schema(&reader);
       let batch = reader.next().unwrap().unwrap();
   
       assert_eq!(reader_schema, batch.schema());
   }
   ```
   
   The assertion fails:
   
   ```text
   reader schema: [a: Int32, b: Int32]
   batch schema:  [b: Int32]
   ```
   
   ### Expected behavior
   
   `RecordBatchReader::schema()` should return the projected schema used by the batches yielded by the reader. In this example, both schemas should contain only field `b`.
   
   ### Additional context
   
   Reproduced on commit [`cc83f7e`](https://github.com/apache/arrow-rs/commit/cc83f7eb25fbf0f6e8f94656d8fd703470e32c04)(`arrow-csv` 59.1.0).
   
   `Reader::schema()` applies the projection, but the `RecordBatchReader` implementation returns `self.decoder.schema.clone()` directly:
   
   https://github.com/apache/arrow-rs/blob/cc83f7eb25fbf0f6e8f94656d8fd703470e32c04/arrow-csv/src/reader/mod.rs#L565-L568
   
   The `RecordBatchReader` implementation was originally added in [#4195](https://github.com/apache/arrow-rs/pull/4195). I could not find an existing issue or PR reporting this specific mismatch.
   
   I would be happy to work on a fix and add regression tests if the maintainers agree with the expected behavior described above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] arrow-csv: RecordBatchReader::schema ignores column projection [arrow-rs]

["Jefffrey (via GitHub)" <gi...@apache.org>]

Jefffrey closed issue #10382: arrow-csv: RecordBatchReader::schema ignores column projection
URL: https://github.com/apache/arrow-rs/issues/10382


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] arrow-csv: RecordBatchReader::schema ignores column projection [arrow-rs]

["Jefffrey (via GitHub)" <gi...@apache.org>]

Jefffrey commented on issue #10382:
URL: https://github.com/apache/arrow-rs/issues/10382#issuecomment-5111797325

   > I would be happy to work on a fix and add regression tests if the maintainers agree with the expected behavior described above.
   
   This sounds reasonable, thanks for the detailed report


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org