You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "psvri (via GitHub)" <gi...@apache.org> on 2025/06/06 18:02:35 UTC

[I] [Proposal] Donate arrow-gpu (a cross platform gpu implementation of arrow using wgpu) to arrow-rs. [arrow-rs]

psvri opened a new issue, #7618:
URL: https://github.com/apache/arrow-rs/issues/7618

   Hello ,
   
   Will the team be interested in taking over the project https://github.com/psvri/arrow-gpu . This is a gpu implementation of apache arrow written in rust using [wgpu](https://github.com/gfx-rs/wgpu) .Honestly I am not able to spend more time on it due to personal life and at the same time I dont want to make the project go to waste. My hope is, if its moved to the apache foundation there will be a lot more people who would contribute and make it successful. Its around 12k lines of rust code and another few k lines of wgsl shaders. 
   
   I have implemented quite a lot of kernels and datatypes . For a list of supported kernels and dataypes you can check this [doc](https://psvri.github.io/arrow-gpu/) .
   
   At this moment its not as performant as I would like it be, but its certainly way faster than CPU kernels. For e.g to incr 10 million non null f32 arrow arrays on my laptop  takes 25% of the time compared to the cpu arrow-rs implementation (This was my first time writing gpu kernels , so perhaps someone more experienced can improve it further). Benchmark code is available [here](https://github.com/psvri/arrow-gpu/blob/main/crates/benchmarks/benches/compare_gpu_arrow.rs) .
   ```
   gpu f32                 time:   [2.4419 ms 2.4573 ms 2.4732 ms]
   cpu f32                 time:   [9.7053 ms 9.7598 ms 9.8199 ms]
   ```
   The primary reason I had created this project was to get a cross platform, vendor independent gpu version of arrow. After researching and tinkering I landed on wgpu since thats the best lowest common denominator across platforms and vendors, and wgpu has very good chance of not being unmaintained given that its gonna be in the web . 
   
   If time had permitted I would have liked to integrate this in other projects like pyarrow, datafusion, polars and apache-spark and even aligned it with https://arrow.apache.org/docs/format/CDeviceDataInterface.html .
   
   I am willing to help with all the efforts needed for the movement.
   
   Sincerely,
   psvri


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Proposal] Donate arrow-gpu (a cross platform gpu implementation of arrow using wgpu) to arrow-rs. [arrow-rs]

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #7618:
URL: https://github.com/apache/arrow-rs/issues/7618#issuecomment-2954382029

   This is pretty cool! But for it to make sense into `arrow-rs`, it would have to somehow build up on the existing APIs.
   
   I couldn't continue on the work, but last year I started refactoring Arrow C++ code to be aware of the device on which the buffers being processed are located [1]. Something similar could be done in `arrow-rs`. To reduce the maintenance burden, most of the GPU kernels themselves could live outside `arrow-rs`, but `arrow-rs` would at least be able to receive arrays stored in GPU memory and pass them to kernels that can process them in the GPU.
   
   [1] https://github.com/apache/arrow/pull/43542


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Proposal] Donate arrow-gpu (a cross platform gpu implementation of arrow using wgpu) to arrow-rs. [arrow-rs]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #7618:
URL: https://github.com/apache/arrow-rs/issues/7618#issuecomment-2950360245

   The project looks really interesting, but we would typically look for an active community before accepting a donation to ensure that development continues after the initial donation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org