[GitHub] [pinot] gortiz commented on issue #9162: Explore alternative to LBuffer for larger than 2GB buffers

[GitBox <gi...@apache.org>]

gortiz commented on issue #9162:
URL: https://github.com/apache/pinot/issues/9162#issuecomment-1327130548

   I've tried Chronicle Bytes in https://github.com/apache/pinot/pull/9842, but I after fixing some errors for a couple of days, I would recommend to do not use it. Some design decision are quite different to what someone using ByteBuffers or LArrays would expect (which is even worse when the methods are not properly documented), some features are not present (like copying to a ByteBuffer from a given offset), some others are not consistent (https://github.com/OpenHFT/Chronicle-Bytes/issues/478), some mistakes derive in the JVM been killed, etc. I think it is an amazing library, but it is not maintained as a generic byte buffer library but very focused on how people from OpenHFT use it, which I totally understand but would require some time inversion from us to be able to use it.
   
   On the other hand, I've been trying to understand why LArray cannot be used in Java 17 and it seems to be related to the way @xerial creates ByteBuffers views. To do so, LArray uses a private constructor that has changed from in Java 17: A new parameter has been added. Xerial has already dealt with that issue [here](https://github.com/msgpack/msgpack-java/commit/117f1612fc3c08741a8acb159e52aa5d29c50988) so I think it is feasible to do it in LArray. That would be fantastic because it would allow us to run with modern JVMs without needing to change a bit of code.
   
   I'm still thinking that a proper Foreign Memory Access API (like JEP-424](https://openjdk.org/jeps/424)) is the way to go, but given that the solution is still not stable, the best we can do is to help Xerial to update his library.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org