Using parquet-cpp and pyarrow

[Renato Marroquín Mogrovejo <re...@gmail.com>]

Hi arrow experts,

I have a c++ application that uses parquet_cpp which I compile using the
folling flags:
LDFLAGS = -larrow -lparquet
which work as expected, i.e. I can read and write parquet files.

I also have a python front-end that invokes this c++ application where I
want to use PyArrow to get some stats about the file before passing those
to my c++ back-end. However, only by importing PyArrow (import pyarrow as
pa) I am getting the following error:

/usr/local/lib/libparquet.so.1: undefined symbol:
_ZN5arrow2io16MemoryMappedFile4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_8FileMode4typeEPSt10shared_ptrIS1_E.

I have installed everything using apt-get install, and the versions I have
are:

pyarrow (0.8.0)
pandas (0.22.0)
parquet (1.2)

Do you guys have any idea on what I am doing wrong?
Thanks in advance!


Renato M.

Re: Using parquet-cpp and pyarrow

[Wes McKinney <we...@gmail.com>]

hi Renato -- how did you install each of the libraries? It looks like
the libraries were not all built with the same compiler toolchain. If
you are using binaries from conda-forge or PyPI, you must either use
gcc 4.8/4.9 or pass -D_GLIBCXX_USE_CXX11_ABI=0 when compiling as Uwe
stated

- Wes

On Tue, Feb 27, 2018 at 4:44 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
> Hello Renato,
>
> the missing symbols says that Parquet is expecting the arrow functionality to be available using the new C++11 ABI. This typically points to the problem that arrow-cpp/parquet-cpp/pyarrow were built using different settings of _GLIBCXX_USE_CXX11_ABI on your system.
>
> As in your case the C++ combination of arrow-cpp and parquet-cpp works, I would expect that in the setting with pyarrow, you have a different version of arrow-cpp also in the path. Thus your system default parquet-cpp also sees a different version of arrow-cpp which is incompatible to it.
>
> The best thing would be to check using ldd which libraries are loading when using pyarrow (e.g. by looking at ldd pyarrow/lib.so).
>
> Uwe
>
> On Tue, Feb 27, 2018, at 9:31 AM, Renato Marroquín Mogrovejo wrote:
>> Hi arrow experts,
>>
>> I have a c++ application that uses parquet_cpp which I compile using the
>> folling flags:
>> LDFLAGS = -larrow -lparquet
>> which work as expected, i.e. I can read and write parquet files.
>>
>> I also have a python front-end that invokes this c++ application where I
>> want to use PyArrow to get some stats about the file before passing those
>> to my c++ back-end. However, only by importing PyArrow (import pyarrow as
>> pa) I am getting the following error:
>>
>> /usr/local/lib/libparquet.so.1: undefined symbol:
>> _ZN5arrow2io16MemoryMappedFile4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_8FileMode4typeEPSt10shared_ptrIS1_E.
>>
>> I have installed everything using apt-get install, and the versions I have
>> are:
>>
>> pyarrow (0.8.0)
>> pandas (0.22.0)
>> parquet (1.2)
>>
>> Do you guys have any idea on what I am doing wrong?
>> Thanks in advance!
>>
>>
>> Renato M.

Re: Using parquet-cpp and pyarrow

["Uwe L. Korn" <uw...@xhochy.com>]

Hello Renato,

the missing symbols says that Parquet is expecting the arrow functionality to be available using the new C++11 ABI. This typically points to the problem that arrow-cpp/parquet-cpp/pyarrow were built using different settings of _GLIBCXX_USE_CXX11_ABI on your system.

As in your case the C++ combination of arrow-cpp and parquet-cpp works, I would expect that in the setting with pyarrow, you have a different version of arrow-cpp also in the path. Thus your system default parquet-cpp also sees a different version of arrow-cpp which is incompatible to it.

The best thing would be to check using ldd which libraries are loading when using pyarrow (e.g. by looking at ldd pyarrow/lib.so).

Uwe

On Tue, Feb 27, 2018, at 9:31 AM, Renato Marroquín Mogrovejo wrote:
> Hi arrow experts,
> 
> I have a c++ application that uses parquet_cpp which I compile using the
> folling flags:
> LDFLAGS = -larrow -lparquet
> which work as expected, i.e. I can read and write parquet files.
> 
> I also have a python front-end that invokes this c++ application where I
> want to use PyArrow to get some stats about the file before passing those
> to my c++ back-end. However, only by importing PyArrow (import pyarrow as
> pa) I am getting the following error:
> 
> /usr/local/lib/libparquet.so.1: undefined symbol:
> _ZN5arrow2io16MemoryMappedFile4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_8FileMode4typeEPSt10shared_ptrIS1_E.
> 
> I have installed everything using apt-get install, and the versions I have
> are:
> 
> pyarrow (0.8.0)
> pandas (0.22.0)
> parquet (1.2)
> 
> Do you guys have any idea on what I am doing wrong?
> Thanks in advance!
> 
> 
> Renato M.