You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Tahsin Hassan <th...@mathworks.com> on 2021/03/05 15:09:27 UTC
Re: [C++] libarrow isolation
Hi,
I was wondering whether folks had a chance to look over the material and had any pointers for the proposed approach.
If I should post in some other format or clarify something, please let me know.
In the meantime, I will try out the steps we propose.
Regards,
Tahsin
From: Tahsin Hassan <th...@mathworks.com>
Date: Thursday, February 25, 2021 at 11:43 AM
To: dev@arrow.apache.org <de...@arrow.apache.org>
Subject: Re: [C++] libarrow isolation
Hi Antoine,
I struggled a bit to put all my thoughts in an email format, that will be easily consumable.
So, I wrote up a github markdown to add some more detail to the issue, we are facing.
Could you take a look, and let us know your thoughts?
https://github.com/mathworks/matlab-arrow-support-files/blob/main/libarrowclash.md
Regards,
Tahsin
From: Antoine Pitrou <an...@python.org>
Date: Tuesday, February 23, 2021 at 1:21 PM
To: dev@arrow.apache.org <de...@arrow.apache.org>
Subject: Re: [C++] libarrow isolation
Hi Tahsin,
I see. So the error happens when loading PyArrow into MATLAB, I
suppose? What kind of error do you get?
Regards
Antoine.
Le 23/02/2021 à 18:12, Tahsin Hassan a écrit :
> Hi Antoine,
>
> MATLAB is using RTLD_GLOBAL. Hope that helps in clarifying the workflow.
>
> Regrards,
> Tahsin
>
> ________________________________
> From: Antoine Pitrou <an...@python.org>
> Sent: Monday, February 22, 2021 9:41 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: [C++] libarrow isolation
>
>
> Le 22/02/2021 à 15:29, Tahsin Hassan a écrit :
>> Hi all,
>>
>> MATLAB uses the Arrow C++ libraries (i.e. libarrow.so) to read and write Parquet files (https://www.mathworks.com/help/matlab/ref/parquetread.html) While exploring ways to integrate more tightly with Arrow, we've run into a symbol/library naming clash issue.
>>
>> When running pyarrow within the MATLAB process (https://www.mathworks.com/help/matlab/call-python-libraries.html?s_tid=CRUX_lftnav ), the libarrow.so loaded by pyarrow clashes with the libarrow.so shipping with MATLAB.
>
> For the record, Python loads extension modules (such as PyArrow) with
> RTLD_LOCAL. I assume MATLAB doesn't?
>
> Regards
>
> Antoine.
>
>
Re: [C++] libarrow isolation
Posted by Antoine Pitrou <an...@python.org>.
There's perhaps a simpler way for MATLAB to solve this:
1. link Arrow statically inside your own libmatlab_arrow.dll
2. have libmatlab_arrow.dll expose its own API corresponding to MATLAB needs
3. arrange for Arrow symbols to not be exposed by libmatlab_arrow.dll
I suppose it depends whether the API needed by MATLAB is small or large.
Regards
Antoine.
Le 10/03/2021 à 04:36, Tahsin Hassan a écrit :
> Hi Wes,
>
> Thanks for providing the feedback.
>
> 1. Regarding the symbol versioning change
> As a next step, should I open up a Jira issue and assign myself to see whether I can automate adding Macros
> BEGIN_ARROW_NS/ END_ARROW_NS ? using LLVM clang AST matchers.
> We can submit this as a one-time mechanical change like you mentioned.
>
> However, “setting up a coding guideline of ensuring `arrow/config.h` by all headers” - - should this also be part of the same issue?
> Is there some other buy-in process for such guidelines?
>
>
>
> 1. There is another issue, of library versioning, which comes even before symbol version issue.
> Since, right now, both pyarrow and MATLAB ships a dll that is aptly named arrow.dll on windows, if pyarrow/python brings in the arrow.dll, the MATLAB process will not load our shipped arrow.dll to begin with due to RTLD_GLOBAL.
>
> We can handle the library versioning issue, by changing some CMake Infrastructure, by giving user ability for specifying their own
> SOVERSION - in linux/mac.
> VERSION/OUTPUT_NAME - in windows (NOT sure yet, just a hunch for windows)
>
> Currently, these cmake variables are located here :
> https://github.com/apache/arrow/blob/0f72bcfc67bb4db72c27f9c3282fe5020490f214/cpp/cmake_modules/BuildUtils.cmake#L369
> Once again, I would be happy to create another Jira issue and see whether I could work through the change.
>
> Regards,
> Tahsin
>
>
> From: Wes McKinney <we...@gmail.com>
> Date: Sunday, March 7, 2021 at 4:12 PM
> To: dev <de...@arrow.apache.org>
> Subject: Re: [C++] libarrow isolation
> I took a look at the document. Basically you want to have two
> different versions of the Arrow shared library loaded into the same
> process, with some code linked to one library and some code linked to
> another. This is very similar to the problem that Boost addresses with
> the `bcp --namespace=$MY_PRIVATE_BOOST_NAMESPACE ...` operation.
>
> To obtain strict symbol isolation you have to change the "arrow::"
> namespace in the C++ libraries you are shipping in your application.
> AFAIK, this is a bit of a nuisance to do. One way to achieve it is to
> replace every use of
>
> namespace arrow {
> ...
> }
>
> with
>
> BEGIN_ARROW_NS
> ...
> END_ARROW_NS
>
> or similar. Then you would do something like `cmake
> -DARROW_NAMESPACE=mwarrow ...` when configuring your build.
>
> This change could certainly be implemented in the Arrow library, which
> is a one-time mechanical operation but will create some ongoing
> non-intuitiveness anytime new files are created. All headers must also
> therefore depend on a central `arrow/config.h` which contains the
> needed namespace macros.
>
> On Fri, Mar 5, 2021 at 9:19 AM Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Hi Tashin,
>>
>> Sorry for the lack of response. Unfortunatly I feel a bit out of my
>> depth on linker issues. I hope someone else can give advice.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 05/03/2021 à 16:09, Tahsin Hassan a écrit :
>>> Hi,
>>>
>>> I was wondering whether folks had a chance to look over the material and had any pointers for the proposed approach.
>>> If I should post in some other format or clarify something, please let me know.
>>> In the meantime, I will try out the steps we propose.
>>>
>>> Regards,
>>> Tahsin
>>>
>>>
>>> From: Tahsin Hassan <th...@mathworks.com>
>>> Date: Thursday, February 25, 2021 at 11:43 AM
>>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>>> Subject: Re: [C++] libarrow isolation
>>> Hi Antoine,
>>>
>>> I struggled a bit to put all my thoughts in an email format, that will be easily consumable.
>>> So, I wrote up a github markdown to add some more detail to the issue, we are facing.
>>>
>>> Could you take a look, and let us know your thoughts?
>>> https://github.com/mathworks/matlab-arrow-support-files/blob/main/libarrowclash.md<https://github.com/mathworks/matlab-arrow-support-files/blob/main/libarrowclash.md>
>>>
>>> Regards,
>>> Tahsin
>>>
>>> From: Antoine Pitrou <an...@python.org>
>>> Date: Tuesday, February 23, 2021 at 1:21 PM
>>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>>> Subject: Re: [C++] libarrow isolation
>>>
>>> Hi Tahsin,
>>>
>>> I see. So the error happens when loading PyArrow into MATLAB, I
>>> suppose? What kind of error do you get?
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> Le 23/02/2021 à 18:12, Tahsin Hassan a écrit :
>>>> Hi Antoine,
>>>>
>>>> MATLAB is using RTLD_GLOBAL. Hope that helps in clarifying the workflow.
>>>>
>>>> Regrards,
>>>> Tahsin
>>>>
>>>> ________________________________
>>>> From: Antoine Pitrou <an...@python.org>
>>>> Sent: Monday, February 22, 2021 9:41 AM
>>>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>>>> Subject: Re: [C++] libarrow isolation
>>>>
>>>>
>>>> Le 22/02/2021 à 15:29, Tahsin Hassan a écrit :
>>>>> Hi all,
>>>>>
>>>>> MATLAB uses the Arrow C++ libraries (i.e. libarrow.so) to read and write Parquet files (https://www.mathworks.com/help/matlab/ref/parquetread.html) While exploring ways to integrate more tightly with Arrow, we've run into a symbol/library naming clash issue.
>>>>>
>>>>> When running pyarrow within the MATLAB process (https://www.mathworks.com/help/matlab/call-python-libraries.html?s_tid=CRUX_lftnav ), the libarrow.so loaded by pyarrow clashes with the libarrow.so shipping with MATLAB.
>>>>
>>>> For the record, Python loads extension modules (such as PyArrow) with
>>>> RTLD_LOCAL. I assume MATLAB doesn't?
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>
>
Re: [C++] libarrow isolation
Posted by Tahsin Hassan <th...@mathworks.com>.
Hi Wes,
Thanks for providing the feedback.
1. Regarding the symbol versioning change
As a next step, should I open up a Jira issue and assign myself to see whether I can automate adding Macros
BEGIN_ARROW_NS/ END_ARROW_NS ? using LLVM clang AST matchers.
We can submit this as a one-time mechanical change like you mentioned.
However, “setting up a coding guideline of ensuring `arrow/config.h` by all headers” - - should this also be part of the same issue?
Is there some other buy-in process for such guidelines?
1. There is another issue, of library versioning, which comes even before symbol version issue.
Since, right now, both pyarrow and MATLAB ships a dll that is aptly named arrow.dll on windows, if pyarrow/python brings in the arrow.dll, the MATLAB process will not load our shipped arrow.dll to begin with due to RTLD_GLOBAL.
We can handle the library versioning issue, by changing some CMake Infrastructure, by giving user ability for specifying their own
SOVERSION - in linux/mac.
VERSION/OUTPUT_NAME - in windows (NOT sure yet, just a hunch for windows)
Currently, these cmake variables are located here :
https://github.com/apache/arrow/blob/0f72bcfc67bb4db72c27f9c3282fe5020490f214/cpp/cmake_modules/BuildUtils.cmake#L369
Once again, I would be happy to create another Jira issue and see whether I could work through the change.
Regards,
Tahsin
From: Wes McKinney <we...@gmail.com>
Date: Sunday, March 7, 2021 at 4:12 PM
To: dev <de...@arrow.apache.org>
Subject: Re: [C++] libarrow isolation
I took a look at the document. Basically you want to have two
different versions of the Arrow shared library loaded into the same
process, with some code linked to one library and some code linked to
another. This is very similar to the problem that Boost addresses with
the `bcp --namespace=$MY_PRIVATE_BOOST_NAMESPACE ...` operation.
To obtain strict symbol isolation you have to change the "arrow::"
namespace in the C++ libraries you are shipping in your application.
AFAIK, this is a bit of a nuisance to do. One way to achieve it is to
replace every use of
namespace arrow {
...
}
with
BEGIN_ARROW_NS
...
END_ARROW_NS
or similar. Then you would do something like `cmake
-DARROW_NAMESPACE=mwarrow ...` when configuring your build.
This change could certainly be implemented in the Arrow library, which
is a one-time mechanical operation but will create some ongoing
non-intuitiveness anytime new files are created. All headers must also
therefore depend on a central `arrow/config.h` which contains the
needed namespace macros.
On Fri, Mar 5, 2021 at 9:19 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hi Tashin,
>
> Sorry for the lack of response. Unfortunatly I feel a bit out of my
> depth on linker issues. I hope someone else can give advice.
>
> Regards
>
> Antoine.
>
>
> Le 05/03/2021 à 16:09, Tahsin Hassan a écrit :
> > Hi,
> >
> > I was wondering whether folks had a chance to look over the material and had any pointers for the proposed approach.
> > If I should post in some other format or clarify something, please let me know.
> > In the meantime, I will try out the steps we propose.
> >
> > Regards,
> > Tahsin
> >
> >
> > From: Tahsin Hassan <th...@mathworks.com>
> > Date: Thursday, February 25, 2021 at 11:43 AM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>
> > Subject: Re: [C++] libarrow isolation
> > Hi Antoine,
> >
> > I struggled a bit to put all my thoughts in an email format, that will be easily consumable.
> > So, I wrote up a github markdown to add some more detail to the issue, we are facing.
> >
> > Could you take a look, and let us know your thoughts?
> > https://github.com/mathworks/matlab-arrow-support-files/blob/main/libarrowclash.md<https://github.com/mathworks/matlab-arrow-support-files/blob/main/libarrowclash.md>
> >
> > Regards,
> > Tahsin
> >
> > From: Antoine Pitrou <an...@python.org>
> > Date: Tuesday, February 23, 2021 at 1:21 PM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>
> > Subject: Re: [C++] libarrow isolation
> >
> > Hi Tahsin,
> >
> > I see. So the error happens when loading PyArrow into MATLAB, I
> > suppose? What kind of error do you get?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 23/02/2021 à 18:12, Tahsin Hassan a écrit :
> >> Hi Antoine,
> >>
> >> MATLAB is using RTLD_GLOBAL. Hope that helps in clarifying the workflow.
> >>
> >> Regrards,
> >> Tahsin
> >>
> >> ________________________________
> >> From: Antoine Pitrou <an...@python.org>
> >> Sent: Monday, February 22, 2021 9:41 AM
> >> To: dev@arrow.apache.org <de...@arrow.apache.org>
> >> Subject: Re: [C++] libarrow isolation
> >>
> >>
> >> Le 22/02/2021 à 15:29, Tahsin Hassan a écrit :
> >>> Hi all,
> >>>
> >>> MATLAB uses the Arrow C++ libraries (i.e. libarrow.so) to read and write Parquet files (https://www.mathworks.com/help/matlab/ref/parquetread.html) While exploring ways to integrate more tightly with Arrow, we've run into a symbol/library naming clash issue.
> >>>
> >>> When running pyarrow within the MATLAB process (https://www.mathworks.com/help/matlab/call-python-libraries.html?s_tid=CRUX_lftnav ), the libarrow.so loaded by pyarrow clashes with the libarrow.so shipping with MATLAB.
> >>
> >> For the record, Python loads extension modules (such as PyArrow) with
> >> RTLD_LOCAL. I assume MATLAB doesn't?
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >
Re: [C++] libarrow isolation
Posted by Wes McKinney <we...@gmail.com>.
I took a look at the document. Basically you want to have two
different versions of the Arrow shared library loaded into the same
process, with some code linked to one library and some code linked to
another. This is very similar to the problem that Boost addresses with
the `bcp --namespace=$MY_PRIVATE_BOOST_NAMESPACE ...` operation.
To obtain strict symbol isolation you have to change the "arrow::"
namespace in the C++ libraries you are shipping in your application.
AFAIK, this is a bit of a nuisance to do. One way to achieve it is to
replace every use of
namespace arrow {
...
}
with
BEGIN_ARROW_NS
...
END_ARROW_NS
or similar. Then you would do something like `cmake
-DARROW_NAMESPACE=mwarrow ...` when configuring your build.
This change could certainly be implemented in the Arrow library, which
is a one-time mechanical operation but will create some ongoing
non-intuitiveness anytime new files are created. All headers must also
therefore depend on a central `arrow/config.h` which contains the
needed namespace macros.
On Fri, Mar 5, 2021 at 9:19 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hi Tashin,
>
> Sorry for the lack of response. Unfortunatly I feel a bit out of my
> depth on linker issues. I hope someone else can give advice.
>
> Regards
>
> Antoine.
>
>
> Le 05/03/2021 à 16:09, Tahsin Hassan a écrit :
> > Hi,
> >
> > I was wondering whether folks had a chance to look over the material and had any pointers for the proposed approach.
> > If I should post in some other format or clarify something, please let me know.
> > In the meantime, I will try out the steps we propose.
> >
> > Regards,
> > Tahsin
> >
> >
> > From: Tahsin Hassan <th...@mathworks.com>
> > Date: Thursday, February 25, 2021 at 11:43 AM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>
> > Subject: Re: [C++] libarrow isolation
> > Hi Antoine,
> >
> > I struggled a bit to put all my thoughts in an email format, that will be easily consumable.
> > So, I wrote up a github markdown to add some more detail to the issue, we are facing.
> >
> > Could you take a look, and let us know your thoughts?
> > https://github.com/mathworks/matlab-arrow-support-files/blob/main/libarrowclash.md
> >
> > Regards,
> > Tahsin
> >
> > From: Antoine Pitrou <an...@python.org>
> > Date: Tuesday, February 23, 2021 at 1:21 PM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>
> > Subject: Re: [C++] libarrow isolation
> >
> > Hi Tahsin,
> >
> > I see. So the error happens when loading PyArrow into MATLAB, I
> > suppose? What kind of error do you get?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 23/02/2021 à 18:12, Tahsin Hassan a écrit :
> >> Hi Antoine,
> >>
> >> MATLAB is using RTLD_GLOBAL. Hope that helps in clarifying the workflow.
> >>
> >> Regrards,
> >> Tahsin
> >>
> >> ________________________________
> >> From: Antoine Pitrou <an...@python.org>
> >> Sent: Monday, February 22, 2021 9:41 AM
> >> To: dev@arrow.apache.org <de...@arrow.apache.org>
> >> Subject: Re: [C++] libarrow isolation
> >>
> >>
> >> Le 22/02/2021 à 15:29, Tahsin Hassan a écrit :
> >>> Hi all,
> >>>
> >>> MATLAB uses the Arrow C++ libraries (i.e. libarrow.so) to read and write Parquet files (https://www.mathworks.com/help/matlab/ref/parquetread.html) While exploring ways to integrate more tightly with Arrow, we've run into a symbol/library naming clash issue.
> >>>
> >>> When running pyarrow within the MATLAB process (https://www.mathworks.com/help/matlab/call-python-libraries.html?s_tid=CRUX_lftnav ), the libarrow.so loaded by pyarrow clashes with the libarrow.so shipping with MATLAB.
> >>
> >> For the record, Python loads extension modules (such as PyArrow) with
> >> RTLD_LOCAL. I assume MATLAB doesn't?
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >
Re: [C++] libarrow isolation
Posted by Antoine Pitrou <an...@python.org>.
Hi Tashin,
Sorry for the lack of response. Unfortunatly I feel a bit out of my
depth on linker issues. I hope someone else can give advice.
Regards
Antoine.
Le 05/03/2021 à 16:09, Tahsin Hassan a écrit :
> Hi,
>
> I was wondering whether folks had a chance to look over the material and had any pointers for the proposed approach.
> If I should post in some other format or clarify something, please let me know.
> In the meantime, I will try out the steps we propose.
>
> Regards,
> Tahsin
>
>
> From: Tahsin Hassan <th...@mathworks.com>
> Date: Thursday, February 25, 2021 at 11:43 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: [C++] libarrow isolation
> Hi Antoine,
>
> I struggled a bit to put all my thoughts in an email format, that will be easily consumable.
> So, I wrote up a github markdown to add some more detail to the issue, we are facing.
>
> Could you take a look, and let us know your thoughts?
> https://github.com/mathworks/matlab-arrow-support-files/blob/main/libarrowclash.md
>
> Regards,
> Tahsin
>
> From: Antoine Pitrou <an...@python.org>
> Date: Tuesday, February 23, 2021 at 1:21 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: [C++] libarrow isolation
>
> Hi Tahsin,
>
> I see. So the error happens when loading PyArrow into MATLAB, I
> suppose? What kind of error do you get?
>
> Regards
>
> Antoine.
>
>
> Le 23/02/2021 à 18:12, Tahsin Hassan a écrit :
>> Hi Antoine,
>>
>> MATLAB is using RTLD_GLOBAL. Hope that helps in clarifying the workflow.
>>
>> Regrards,
>> Tahsin
>>
>> ________________________________
>> From: Antoine Pitrou <an...@python.org>
>> Sent: Monday, February 22, 2021 9:41 AM
>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>> Subject: Re: [C++] libarrow isolation
>>
>>
>> Le 22/02/2021 à 15:29, Tahsin Hassan a écrit :
>>> Hi all,
>>>
>>> MATLAB uses the Arrow C++ libraries (i.e. libarrow.so) to read and write Parquet files (https://www.mathworks.com/help/matlab/ref/parquetread.html) While exploring ways to integrate more tightly with Arrow, we've run into a symbol/library naming clash issue.
>>>
>>> When running pyarrow within the MATLAB process (https://www.mathworks.com/help/matlab/call-python-libraries.html?s_tid=CRUX_lftnav ), the libarrow.so loaded by pyarrow clashes with the libarrow.so shipping with MATLAB.
>>
>> For the record, Python loads extension modules (such as PyArrow) with
>> RTLD_LOCAL. I assume MATLAB doesn't?
>>
>> Regards
>>
>> Antoine.
>>
>>
>