You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jeroen (via GitHub)" <gi...@apache.org> on 2023/12/04 19:17:03 UTC

[I] Consider merging static libarrow builds into universal binaries [arrow]

jeroen opened a new issue, #39068:
URL: https://github.com/apache/arrow/issues/39068

   ### Describe the enhancement requested
   
   When building the R package on MacOS it automatically downloads static libarrow builds based on the local host arch:
   
    - https://nightlies.apache.org/arrow/r/libarrow/bin/darwin-arm64-openssl-1.1/
    - https://nightlies.apache.org/arrow/r/libarrow/bin/darwin-x86_64-openssl-1.1/
   
   
   However the host arch is sometimes different from the target, for example when we build on an M1 machine targeting intel using `clang -arch x86_64` or the other way around. Posit even compiles their `darwin-arm64` binaries on `linux-arm64`.
   
   Furthermore, R might support universal packages soon that get compiled for both architectures.
   
   It would be safer to merge static libs for arm64 and x86_64 into a single universal ("fat") static library that the R package can download without any guessing of the target arch. You can merge static libs easily using lipo, e.g:
   
   ```sh
   lipo -create libarrow-arm64.a libarrow-x86_64.a -output libarrow-universal.a
   ```
   
   This way you can publish a universal static libarrow that will work both when targeting either arm64 or x86_64, or both.
   
   We also do this for the autobrew libs, and it works very well: https://github.com/autobrew/bundler
   
   
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Consider merging static libarrow builds into universal binaries [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #39068:
URL: https://github.com/apache/arrow/issues/39068#issuecomment-3921986460

   This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Consider merging static libarrow builds into universal binaries [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #39068: Consider merging static libarrow builds into universal binaries
URL: https://github.com/apache/arrow/issues/39068


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Consider merging static libarrow builds into universal binaries [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #39068:
URL: https://github.com/apache/arrow/issues/39068#issuecomment-4620250248

   This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Consider merging static libarrow builds into universal binaries [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39068:
URL: https://github.com/apache/arrow/issues/39068#issuecomment-1840198358

   > IIRC we use universal binaries for the pyarrow wheels as well.
   
   We don't (or at least not anymore, we stopped doing that last year), but I don't think that needs to influence any decision on the R side (there might be different trade offs). 
   In general, in the PyData ecosystem, most projects I know stopped providing the universal wheels, and see this short email thread about it: https://lists.apache.org/thread/6qozocz80wrg3r98zr1dzp7spgvwrvh4


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Consider merging static libarrow builds into universal binaries [arrow]

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #39068:
URL: https://github.com/apache/arrow/issues/39068#issuecomment-1839469637

   Yeah makes sense, we do something similar with windows  and ucrt/non-ucrt after all. IIRC we use universal binaries for the pyarrow wheels as well. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Consider merging static libarrow builds into universal binaries [arrow]

Posted by "jeroen (via GitHub)" <gi...@apache.org>.
jeroen commented on issue #39068:
URL: https://github.com/apache/arrow/issues/39068#issuecomment-1840466454

   Just to be clear, the R packages themselves won't have universal binaries for now, that would be a big change.
   
   The suggestion is merely that the static libarrow bundles are merged into multilibs, which is a small change that will make the build process for R packages simpler and more robust because it won't have to guess the target arch at build time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org