[PR] [coalesce] Specialized InProgressFixedSizeListArray [arrow-rs]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid opened a new pull request, #11120:
URL: https://github.com/apache/arrow-rs/pull/11120

   ## What
   
   Adds `InProgressFixedSizeListArray`, a specialized `InProgressArray` for `FixedSizeList(T, N)` columns in `BatchCoalescer`.
   
   ## Why
   
   Previously, `FixedSizeList` went through `GenericInProgressArray` → `concat_fallback` → `MutableArrayData`. The MutableArrayData path recursively extends child arrays also through MutableArrayData, even when the child type (e.g. `Float32`) has a faster specialized path.
   
   This PR:
   1. Builds the outer null bitmap incrementally via `NullBufferBuilder`
   2. Buffers child value slices (zero-copy from the source array for the non-filter path)
   3. At `finish()`, calls `concat()` on the child slices, which dispatches to the type-optimized path (e.g. `concat_primitives` for `Float32` child arrays)
   
   Common use case: embedding vectors (`FixedSizeList<Float32, 128>`) used in ML pipelines; also geospatial coords (size=2/3/4).
   
   ## Benchmarks
   
   Run with `cargo bench --bench coalesce_kernels --features test_utils -- fsl`.
   
   | Benchmark | Before | After | Speedup |
   |---|---|---|---|
   | fsl(4), no nulls, sel=0.001 | 81.1 ms | 63.3 ms | **−22%** |
   | fsl(128), no nulls, sel=0.001 | 97.5 ms | 76.0 ms | **−22%** |
   | fsl(4), no nulls, sel=0.01 | 11.2 ms | 9.2 ms | **−18%** |
   | fsl(128), no nulls, sel=0.01 | 21.9 ms | 19.1 ms | **−13%** |
   | fsl(4), no nulls, sel=0.1 | 5.3 ms | 4.8 ms | **−10%** |
   | fsl(128), no nulls, sel=0.1 | 16.1 ms | 13.5 ms | **−16%** |
   | fsl(4), no nulls, sel=0.8 | 1.31 ms | 1.16 ms | **−12%** |
   | fsl(128), no nulls, sel=0.8 | 9.2 ms | 8.0 ms | **−13%** |
   | fsl(4), nulls=0.1, sel=0.001 | 180.6 ms | 73.3 ms | **−59%** |
   | fsl(128), nulls=0.1, sel=0.001 | 117.2 ms | 84.9 ms | **−27%** |
   | fsl(4), nulls=0.1, sel=0.1 | 7.5 ms | 6.5 ms | **−14%** |
   | fsl(128), nulls=0.1, sel=0.1 | 56.3 ms | 16.7 ms | **−70%** |
   | fsl(4), nulls=0.1, sel=0.8 | 4.4 ms | 1.72 ms | **−61%** |
   | fsl(128), nulls=0.1, sel=0.8 | 15.9 ms | 9.4 ms | **−41%** |
   
   The nullable cases see the largest gains (up to 70%) because the old path was hitting a particularly slow codepath through MutableArrayData when nulls were present.
   
   Part of https://github.com/apache/arrow-rs/issues/7761


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [coalesce] Specialized InProgressFixedSizeListArray [arrow-rs]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #11120:
URL: https://github.com/apache/arrow-rs/pull/11120#issuecomment-5704682848

   Closing — keeping local for now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [coalesce] Specialized InProgressFixedSizeListArray [arrow-rs]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid closed pull request #11120: [coalesce] Specialized InProgressFixedSizeListArray
URL: https://github.com/apache/arrow-rs/pull/11120


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org