You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pchintar (via GitHub)" <gi...@apache.org> on 2026/05/08 19:16:11 UTC

[I] IPC reader projection does not handle duplicate projection indices correctly [arrow-rs]

pchintar opened a new issue, #9950:
URL: https://github.com/apache/arrow-rs/issues/9950

   ### Description
   
   Currently, when reading IPC data with column projection enabled, duplicate projection indices can produce an invalid `RecordBatch`.
   
   ---
   
   ### Root Cause
   
   In `arrow-ipc/src/reader.rs`, projected columns are matched using:
   
   ```rust id="jlwmjv"
   projection.iter().position(|p| p == &idx)
   ```
   
   However, `position()` only returns the first matching entry.
   
   For example:
   
   ```rust id="7bgmrr"
   projection = vec![1, 1]
   ```
   
   Only a single column is decoded even though the projected schema contains two fields.
   
   `Schema::project` and `RecordBatch::project` both allow duplicate projection indices, so the IPC reader behavior becomes inconsistent with the rest of Arrow.
   
   ---
   
   ### Impact
   
   Can lead to:
   
   * invalid `RecordBatch` construction
   * runtime errors due to schema/column count mismatch
   
   Occurs when:
   
   * projection contains duplicate indices
   * reading IPC data through `FileReader` or `StreamReader`
   
   ---
   
   ### Reproduction
   
   A minimal example:
   
   ```rust id="s4q7gx"
   let projection = vec![1, 1];
   
   let reader =
       FileReader::try_new(std::io::Cursor::new(buf), Some(projection))?;
   ```
   
   Before fix:
   
   ```text id="uwmkmt"
   InvalidArgumentError(
       "number of columns(1) must match number of fields(2) in schema"
   )
   ```
   
   ---
   
   ### Proposed Fix to this Bug
   
   Update projection handling to preserve all matching projection entries while decoding each physical field only once.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] IPC reader projection does not handle duplicate projection indices correctly [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9950:
URL: https://github.com/apache/arrow-rs/issues/9950#issuecomment-4616723510

   `label_issue.py` automatically added labels {'arrow'} from #9952


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] IPC reader projection does not handle duplicate projection indices correctly [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed issue #9950: IPC reader projection does not handle duplicate projection indices correctly
URL: https://github.com/apache/arrow-rs/issues/9950


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org