You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2024/02/29 17:51:16 UTC
[I] Explore MUSL to avoid GLIBC requirement [arrow-datafusion-comet]
sunchao opened a new issue, #138:
URL: https://github.com/apache/arrow-datafusion-comet/issues/138
### What is the problem the feature request solves?
Currently we require dynamically linked glibc on the host machine which isn't desirable since it requires certain version of glibc, otherwise the native Comet lib won't be able to load. In particular, the glibc version used to compile and build Comet must be compatible (i.e., older) than the version used to execute Comet.
Another option is to explore `x86_64-unknown-linux-musl` compile target, which statically links against the [MUSL libc](https://musl.libc.org/) and can effectively remove the above constraints. However, t[here](https://andygrove.io/2020/05/why-musl-extremely-slow/) are some reports that MUSL can cause performance degradation, see [here](https://www.reddit.com/r/rust/comments/oh2k8l/rust_musl_and_glibc_in_2021/) and here. We may need to do some benchmark to evaluate.
### Describe the potential solution
Evaluate MUSL and check if it can be a potential candidate to co-exist (or replace) with the current GLIBC approach.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Explore MUSL to avoid GLIBC requirement [arrow-datafusion-comet]
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #138:
URL: https://github.com/apache/arrow-datafusion-comet/issues/138#issuecomment-1973062125
> In particular, the glibc version used to compile and build Comet must be compatible (i.e., older) than the version used to execute Comet.
In the native world, it usually implies to build and release the dynamic lib on a decent old enough Linux distro. In that way, the built lib could be executed on newer Linux hosts. That' pretty much the simplest way to archive compatibility. And maybe we can reuse rustc's docker image to target lower glibc: https://github.com/rust-lang/rust/blob/master/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Explore MUSL to avoid GLIBC requirement [arrow-datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #138:
URL: https://github.com/apache/arrow-datafusion-comet/issues/138#issuecomment-2016289066
This recent blog post is worth a read. It talks about some performance issues with musl's default allocator and how switching to jemalloc helps.
https://blog.sdf.com/p/fast-development-in-rust-part-one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Explore MUSL to avoid GLIBC requirement [arrow-datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on issue #138:
URL: https://github.com/apache/arrow-datafusion-comet/issues/138#issuecomment-1973506910
Yes, this ticket belongs to the "good to have" category IMO. One disadvantage of having to stick to the lowest common denominator for GLIBC is that certain crates may not able to compile. For instance, we ran into issues with `mimalloc` when building on GLIBC 2.17 IIRC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Explore MUSL to avoid GLIBC requirement [arrow-datafusion-comet]
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #138:
URL: https://github.com/apache/arrow-datafusion-comet/issues/138#issuecomment-2046758883
hhhm, I recently know there's a project called https://github.com/rust-cross/cargo-zigbuild which is able to cross compile rust project to different target including various glibc versions by using zig as a linker. I think we can explore this option to cross compile to low version glibc.
Ref: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html see the glibc part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Explore MUSL to avoid GLIBC requirement [arrow-datafusion-comet]
Posted by "snmvaughan (via GitHub)" <gi...@apache.org>.
snmvaughan commented on issue #138:
URL: https://github.com/apache/arrow-datafusion-comet/issues/138#issuecomment-1973436191
I tend to agree with using a lower target glibc for compatibility.
If we want to consider MUSL we should also plan on benchmarking different combinations of target (glibc vs musl) and alloc (jemalloc vs system alloc) using a benchmark like TPC-DS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Explore MUSL to avoid GLIBC requirement [arrow-datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #138:
URL: https://github.com/apache/arrow-datafusion-comet/issues/138#issuecomment-2016304570
I remember we used jemalloc at the beginning, but we encountered some libraries issues from it and switched it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org