[GitHub] [arrow-rs] jhorstmann opened a new issue #1516: Allow creating buffers from externally owned memory like Vec or String

[GitBox <gi...@apache.org>]

jhorstmann opened a new issue #1516:
URL: https://github.com/apache/arrow-rs/issues/1516


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   I'd like to process data that is coming from outside arrow using arrow compute kernels without copying the data.
   
   The FFI api does not seem to support this use case at the moment. The only way I found to create an `FFI_ArrowArray` was with an already existing `ArrayData` struct. Creating `ArrayData` requires at least one `Buffer`, and creating a `Buffer` from foreign memory again needs an `FFI_ArrowArray`.
   
   **Describe the solution you'd like**
   
   The `Buffer` already supports two ways of deallocating memory.
   
    - Via arrows cache-aligned allocator
    - With a callback in an `FFI_ArrowArray`
   
   The second option can be generalized so that the callback can be any type using a dyn trait.
   
   **Describe alternatives you've considered**
   
   An alternative could be to make it easier to create an `FFI_ArrowArray` from a single memory region, but that seems to be more complex and also less general.
   
   **Additional context**
   
   This functionality could also allow zero-copy conversion from arrow2.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org