You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Will Jones <wi...@voltrondata.com> on 2022/02/18 18:48:23 UTC
RPATH and Brew on MacOS
Is there a recommended way to handle RPATH and brew on MacOS? When trying
to link to libparquet it fails with:
dyld[15311]: Library not loaded: @rpath/libparquet.700.dylib
Referenced from:
/opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/libarrow_dataset.700.0.0.dylib
Reason: tried: '/usr/local/lib/libparquet.700.dylib' (no such file),
'/usr/lib/libparquet.700.dylib' (no such file)Library not loaded:
@rpath/libarrow.700.dylib
Unless I pass in -rpath,/opt/homebrew/opt/apache-arrow/lib/. Is that
expected? Or should I be adding that permanently to my rpath somehow? Or
should these libraries be symlinked to an existing directory on my rpath?
Thanks,
Will Jones
Re: RPATH and Brew on MacOS
Posted by Jonathan Keane <jk...@gmail.com>.
Hello,
> But it sounds like the general process in using the homebrew Arrow binaries on Mac OS is:
>
> 1. brew install apache-arrow
> 2. Either:
> a. Do some rpath modification (like I did): CXX_FLAGS=-Wl,-rpath,/opt/homebrew/opt/apache-arrow/lib/
> b. Or set DYLD_LIBRARY_PATH (as Vibhatha suggested): DYLD_LIBRARY_PATH=/opt/homebrew/opt/apache-arrow/lib/
> 3. g++ $(pkg-config --cflags --libs arrow-flight) $CXX_FLAGS example.cpp -o example
I agree that step (2) here really shouldn't be necessary (and as far
as I remember, hadn't been necessary for previous versions of Arrow).
Is it possible that something changed with the way that libparquet is
being built that it's not being passed the flag to disable rpath that
is set in the homebrew formula [1]?
[1] https://github.com/Homebrew/homebrew-core/blob/f68248b85f8a441b1c5777f2ddcf576ea669c906/Formula/apache-arrow.rb#L73
-Jon
On Tue, Feb 22, 2022 at 9:59 AM Will Jones <wi...@voltrondata.com> wrote:
>
> Thanks for your replies.
>
> > Is your goal to have libarrow be loaded from a relative path of libparquet?
>
> My impression is that I don't have a choice if I installed from homebrew. That @rpath/libarrow.700.dylib reference looks like it's hard-coded in the libparquet binary.
>
> > maybe it is connected to https://github.com/apache/arrow/pull/11602.
>
> Yes, it is turned off in the homebrew build:
> ❯ cat /opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/cmake/arrow/ArrowOptions.cmake | grep RPATH
> set(ARROW_RPATH_ORIGIN "OFF")
> set(ARROW_INSTALL_NAME_RPATH "OFF")
>
> But it sounds like the general process in using the homebrew Arrow binaries on Mac OS is:
>
> 1. brew install apache-arrow
> 2. Either:
> a. Do some rpath modification (like I did): CXX_FLAGS=-Wl,-rpath,/opt/homebrew/opt/apache-arrow/lib/
> b. Or set DYLD_LIBRARY_PATH (as Vibhatha suggested): DYLD_LIBRARY_PATH=/opt/homebrew/opt/apache-arrow/lib/
> 3. g++ $(pkg-config --cflags --libs arrow-flight) $CXX_FLAGS example.cpp -o example
>
> I was wondering if that second step wasn't necessary, but it sounds like it is.
>
>
> On Mon, Feb 21, 2022 at 12:22 AM Alenka Frim <al...@voltrondata.com> wrote:
>>
>> Hi Will,
>>
>> maybe it is connected to https://github.com/apache/arrow/pull/11602.
>>
>> Alenka
>>
>> On Sat, Feb 19, 2022 at 8:18 AM James Duong <ja...@bitquilltech.com> wrote:
>>>
>>> Hi Will,
>>>
>>> Is your goal to have libarrow be loaded from a relative path of libparquet? I've found that @loader_path works well for this and is close to how the rpath works in Linux.
>>> Details about different dynamic loading paths here: https://wincent.com/wiki/%40executable_path%2C_%40load_path_and_%40rpath
>>>
>>> On Fri, Feb 18, 2022 at 5:17 PM Vibhatha Abeykoon <vi...@gmail.com> wrote:
>>>>
>>>> Hello Will,
>>>>
>>>> How about setting the DYLD_LIBRARY_PATH?
>>>>
>>>> On Sat, Feb 19, 2022 at 12:18 AM Will Jones <wi...@voltrondata.com> wrote:
>>>>>
>>>>> Is there a recommended way to handle RPATH and brew on MacOS? When trying to link to libparquet it fails with:
>>>>>
>>>>> dyld[15311]: Library not loaded: @rpath/libparquet.700.dylib
>>>>> Referenced from: /opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/libarrow_dataset.700.0.0.dylib
>>>>> Reason: tried: '/usr/local/lib/libparquet.700.dylib' (no such file), '/usr/lib/libparquet.700.dylib' (no such file)Library not loaded: @rpath/libarrow.700.dylib
>>>>>
>>>>> Unless I pass in -rpath,/opt/homebrew/opt/apache-arrow/lib/. Is that expected? Or should I be adding that permanently to my rpath somehow? Or should these libraries be symlinked to an existing directory on my rpath?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Will Jones
>>>>
>>>> --
>>>> Vibhatha Abeykoon
>>>
>>>
>>>
>>> --
>>> James Duong
>>> Lead Software Developer
>>> Bit Quill Technologies Inc.
>>> Direct: +1.604.562.6082 | jamesd@bitquilltech.com
>>> https://www.bitquilltech.com
>>>
>>> This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.
Re: RPATH and Brew on MacOS
Posted by Will Jones <wi...@voltrondata.com>.
Thanks for your replies.
> Is your goal to have libarrow be loaded from a relative path of
libparquet?
My impression is that I don't have a choice if I installed from homebrew.
That @rpath/libarrow.700.dylib reference looks like it's hard-coded in the
libparquet binary.
> maybe it is connected to https://github.com/apache/arrow/pull/11602.
Yes, it is turned off in the homebrew build:
❯ cat
/opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/cmake/arrow/ArrowOptions.cmake
| grep RPATH
set(ARROW_RPATH_ORIGIN "OFF")
set(ARROW_INSTALL_NAME_RPATH "OFF")
But it sounds like the general process in using the homebrew Arrow binaries
on Mac OS is:
1. brew install apache-arrow
2. Either:
a. Do some rpath modification (like I did):
CXX_FLAGS=-Wl,-rpath,/opt/homebrew/opt/apache-arrow/lib/
b. Or set DYLD_LIBRARY_PATH (as Vibhatha suggested):
DYLD_LIBRARY_PATH=/opt/homebrew/opt/apache-arrow/lib/
3. g++ $(pkg-config --cflags --libs arrow-flight) $CXX_FLAGS example.cpp
-o example
I was wondering if that second step wasn't necessary, but it sounds like it
is.
On Mon, Feb 21, 2022 at 12:22 AM Alenka Frim <al...@voltrondata.com> wrote:
> Hi Will,
>
> maybe it is connected to https://github.com/apache/arrow/pull/11602.
>
> Alenka
>
> On Sat, Feb 19, 2022 at 8:18 AM James Duong <ja...@bitquilltech.com>
> wrote:
>
>> Hi Will,
>>
>> Is your goal to have libarrow be loaded from a relative path of
>> libparquet? I've found that @loader_path works well for this and is close
>> to how the rpath works in Linux.
>> Details about different dynamic loading paths here:
>> https://wincent.com/wiki/%40executable_path%2C_%40load_path_and_%40rpath
>>
>> On Fri, Feb 18, 2022 at 5:17 PM Vibhatha Abeykoon <vi...@gmail.com>
>> wrote:
>>
>>> Hello Will,
>>>
>>> How about setting the DYLD_LIBRARY_PATH?
>>>
>>> On Sat, Feb 19, 2022 at 12:18 AM Will Jones <wi...@voltrondata.com>
>>> wrote:
>>>
>>>> Is there a recommended way to handle RPATH and brew on MacOS? When
>>>> trying to link to libparquet it fails with:
>>>>
>>>> dyld[15311]: Library not loaded: @rpath/libparquet.700.dylib
>>>> Referenced from:
>>>> /opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/libarrow_dataset.700.0.0.dylib
>>>> Reason: tried: '/usr/local/lib/libparquet.700.dylib' (no such file),
>>>> '/usr/lib/libparquet.700.dylib' (no such file)Library not loaded:
>>>> @rpath/libarrow.700.dylib
>>>>
>>>> Unless I pass in -rpath,/opt/homebrew/opt/apache-arrow/lib/. Is that
>>>> expected? Or should I be adding that permanently to my rpath somehow? Or
>>>> should these libraries be symlinked to an existing directory on my rpath?
>>>>
>>>> Thanks,
>>>>
>>>> Will Jones
>>>>
>>> --
>>> Vibhatha Abeykoon
>>>
>>
>>
>> --
>>
>> *James Duong*
>> Lead Software Developer
>> Bit Quill Technologies Inc.
>> Direct: +1.604.562.6082 | jamesd@bitquilltech.com
>> https://www.bitquilltech.com
>>
>> This email message is for the sole use of the intended recipient(s) and
>> may contain confidential and privileged information. Any unauthorized
>> review, use, disclosure, or distribution is prohibited. If you are not the
>> intended recipient, please contact the sender by reply email and destroy
>> all copies of the original message. Thank you.
>>
>
Re: RPATH and Brew on MacOS
Posted by Alenka Frim <al...@voltrondata.com>.
Hi Will,
maybe it is connected to https://github.com/apache/arrow/pull/11602.
Alenka
On Sat, Feb 19, 2022 at 8:18 AM James Duong <ja...@bitquilltech.com> wrote:
> Hi Will,
>
> Is your goal to have libarrow be loaded from a relative path of
> libparquet? I've found that @loader_path works well for this and is close
> to how the rpath works in Linux.
> Details about different dynamic loading paths here:
> https://wincent.com/wiki/%40executable_path%2C_%40load_path_and_%40rpath
>
> On Fri, Feb 18, 2022 at 5:17 PM Vibhatha Abeykoon <vi...@gmail.com>
> wrote:
>
>> Hello Will,
>>
>> How about setting the DYLD_LIBRARY_PATH?
>>
>> On Sat, Feb 19, 2022 at 12:18 AM Will Jones <wi...@voltrondata.com> wrote:
>>
>>> Is there a recommended way to handle RPATH and brew on MacOS? When
>>> trying to link to libparquet it fails with:
>>>
>>> dyld[15311]: Library not loaded: @rpath/libparquet.700.dylib
>>> Referenced from:
>>> /opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/libarrow_dataset.700.0.0.dylib
>>> Reason: tried: '/usr/local/lib/libparquet.700.dylib' (no such file),
>>> '/usr/lib/libparquet.700.dylib' (no such file)Library not loaded:
>>> @rpath/libarrow.700.dylib
>>>
>>> Unless I pass in -rpath,/opt/homebrew/opt/apache-arrow/lib/. Is that
>>> expected? Or should I be adding that permanently to my rpath somehow? Or
>>> should these libraries be symlinked to an existing directory on my rpath?
>>>
>>> Thanks,
>>>
>>> Will Jones
>>>
>> --
>> Vibhatha Abeykoon
>>
>
>
> --
>
> *James Duong*
> Lead Software Developer
> Bit Quill Technologies Inc.
> Direct: +1.604.562.6082 | jamesd@bitquilltech.com
> https://www.bitquilltech.com
>
> This email message is for the sole use of the intended recipient(s) and
> may contain confidential and privileged information. Any unauthorized
> review, use, disclosure, or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by reply email and destroy
> all copies of the original message. Thank you.
>
Re: RPATH and Brew on MacOS
Posted by James Duong <ja...@bitquilltech.com>.
Hi Will,
Is your goal to have libarrow be loaded from a relative path of libparquet?
I've found that @loader_path works well for this and is close to how the
rpath works in Linux.
Details about different dynamic loading paths here:
https://wincent.com/wiki/%40executable_path%2C_%40load_path_and_%40rpath
On Fri, Feb 18, 2022 at 5:17 PM Vibhatha Abeykoon <vi...@gmail.com>
wrote:
> Hello Will,
>
> How about setting the DYLD_LIBRARY_PATH?
>
> On Sat, Feb 19, 2022 at 12:18 AM Will Jones <wi...@voltrondata.com> wrote:
>
>> Is there a recommended way to handle RPATH and brew on MacOS? When trying
>> to link to libparquet it fails with:
>>
>> dyld[15311]: Library not loaded: @rpath/libparquet.700.dylib
>> Referenced from:
>> /opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/libarrow_dataset.700.0.0.dylib
>> Reason: tried: '/usr/local/lib/libparquet.700.dylib' (no such file),
>> '/usr/lib/libparquet.700.dylib' (no such file)Library not loaded:
>> @rpath/libarrow.700.dylib
>>
>> Unless I pass in -rpath,/opt/homebrew/opt/apache-arrow/lib/. Is that
>> expected? Or should I be adding that permanently to my rpath somehow? Or
>> should these libraries be symlinked to an existing directory on my rpath?
>>
>> Thanks,
>>
>> Will Jones
>>
> --
> Vibhatha Abeykoon
>
--
*James Duong*
Lead Software Developer
Bit Quill Technologies Inc.
Direct: +1.604.562.6082 | jamesd@bitquilltech.com
https://www.bitquilltech.com
This email message is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure, or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy
all copies of the original message. Thank you.
Re: RPATH and Brew on MacOS
Posted by Vibhatha Abeykoon <vi...@gmail.com>.
Hello Will,
How about setting the DYLD_LIBRARY_PATH?
On Sat, Feb 19, 2022 at 12:18 AM Will Jones <wi...@voltrondata.com> wrote:
> Is there a recommended way to handle RPATH and brew on MacOS? When trying
> to link to libparquet it fails with:
>
> dyld[15311]: Library not loaded: @rpath/libparquet.700.dylib
> Referenced from:
> /opt/homebrew/Cellar/apache-arrow/7.0.0_1/lib/libarrow_dataset.700.0.0.dylib
> Reason: tried: '/usr/local/lib/libparquet.700.dylib' (no such file),
> '/usr/lib/libparquet.700.dylib' (no such file)Library not loaded:
> @rpath/libarrow.700.dylib
>
> Unless I pass in -rpath,/opt/homebrew/opt/apache-arrow/lib/. Is that
> expected? Or should I be adding that permanently to my rpath somehow? Or
> should these libraries be symlinked to an existing directory on my rpath?
>
> Thanks,
>
> Will Jones
>
--
Vibhatha Abeykoon