[GitHub] [arrow] zeroshade commented on issue #34895: Golang arenas

["zeroshade (via GitHub)" <gi...@apache.org>]

zeroshade commented on issue #34895:
URL: https://github.com/apache/arrow/issues/34895#issuecomment-1497721445

   I think it's a great idea to try using an arena. To my knowledge there hasn't been any attempts yet, and I know that I've gotten at least one question about them, but that's all the interest I've seen so far. There's a few areas I can think of that might be useful for it:
   
   1. An implementation of the `memory.Allocator` interface that uses an arena under the hood for the Allocate, Reallocate, etc. methods for arrays. (this is probably the easiest thing to implement)
   2. An arena in the compute package that we can use for all the small allocations that are used intermittently during the compute execution
   3. As an option for IPC read/write and/or Flight/FlightSQL along with in the Parquet reader and writer.
   4. As an option for the CSV and JSON readers/writers
   
   There's probably other areas that could benefit that I'm not thinking of offhand. 
   
   The one thing to keep in mind is that we do want to try to maintain compatibility with the older versions of go (ie go1.17) so you'd need to keep anything with arenas in files that are protected with a build constraint (for example, notice that all the files in the compute package have `//go:build go1.18` restricting them to only be used when compiling with go1.18+). So that does make things a little complicated, and is why I mentioned that an arena based `memory.Allocator` is likely the easiest thing to implement first.
   
   If you do take a stab at it, I'll happily review the PRs and feel free to reach out if you need any assistance implementing it. Thanks in advance if you do take this on!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org