You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Joseph Porter <jo...@falkonry.com.INVALID> on 2022/10/10 17:27:46 UTC

Build Frustrations

If this isn't the right place to ask, please let me know where to direct
this question:

I'm trying to build a slightly modified version of the 9.0.0 tag for arrow
(particularly Pyarrow).  Walking through the build steps (below), I have
run into a problem where the arrow c++ library has symbols that include the
tag Bcxx11, but the cython libraries aren't generating that flag.  The
symbol resolution fails at runtime once the library loads.

Particularly these two symbols in python/build/lib.linux-x86_64-3.8/pyarrow/
lib.cpython-38-x86_64-linux-gnu.so

U _ZNK5arrow8DataType18ComputeFingerprintEv
U _ZNK5arrow8DataType26ComputeMetadataFingerprintEv

Don't resolve.  All of the corresponding symbols in libarrow_so.900 look
like this (not the addition of the B5cxx11Ev):

00000000006addc0 T _ZNK5arrow8DataType18ComputeFingerprintB5cxx11Ev
00000000006ae050 T _ZNK5arrow8DataType26ComputeMetadataFingerprintB5cxx11Ev

I've tried building with explicit flags to encourage the libraries to
include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't seem to
impact this issue:

set (CMAKE_CXX_STANDARD 11)
set (CMAKE_CXX_STANDARD_REQUIRED ON)
set (CMAKE_CXX_EXTENSIONS OFF)

I also added
export PYARROW_CXXFLAGS="-std=c++11"
for the wheel build of pyarrow (no effect).

I would appreciate any guidance on how to build a clean wheel file of
pyarrow that includes my small code changes.

Best,
-Joe

------------------------------------------------------------------------------------------
Here's what I'm using:
gcc version 9.4.0
g++ version 9.4.0
cmake version 3.16.3

CMake command for cpp/build:
cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib
-DCMAKE_BUILD_TYPE=Release -DARROW_DATASET=ON -DARROW_WITH_BZ2=ON
-DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON -DARROW_WITH_LZ4=ON
-DARROW_WITH_SNAPPY=ON -DARROW_WITH_BROTLI=ON -DARROW_PARQUET=ON
-DPARQUET_REQUIRE_ENCRYPTION=ON -DARROW_PYTHON=ON -DARROW_BUILD_TESTS=ON
-DARROW_CUDA=ON -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON -DARROW_PLASMA=ON
-DARROW_S3=ON -DARROW_TENSORFLOW=ON -DARROW_CSV=ON -DARROW_JSON=ON
-DARROW_WITH_RE2=ON -DARROW_IPC=ON -DARROW_DEPENDENCY_SOURCE=BUNDLED ..

Setup and build for python:
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_BZ2=1
export PYARROW_WITH_ZLIB=1
export PYARROW_WITH_ZSTD=1
export PYARROW_WITH_LZ4=1
export PYARROW_WITH_SNAPPY=1
export PYARROW_WITH_BROTLI=1
export PYARROW_WITH_PARQUET=1
export PYARROW_WITH_PARQUET_ENCRYPTION-1
export PYARROW_WITH_PARQUET_ENCRYPTION=1
export PYARROW_WITH_CUDA=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
export PYARROW_WITH_PLASMA=1
export PYARROW_WITH_S3=1
export PYARROW_WITH_TENSORFLOW=1
export PYARROW_WITH_CSV=1
export PYARROW_WITH_JSON=1
export_PYARROW_WITH_RE2=1
export PYARROW_WITH_IPC=1
export PYARROW_PARALLEL=4
export PYARROW_CXXFLAGS="-std=c++11"
python setup.py build_ext --build-type=release --bundle-arrow-cpp
bdist_wheel

Re: Build Frustrations

Posted by Joseph Porter <jo...@falkonry.com.INVALID>.
Building everything without c++11 seems to work. Thanks again for your help.

On Mon, Oct 10, 2022 at 4:52 PM Joseph Porter <jo...@falkonry.com>
wrote:

> I think I found a culprit in the pyarrow cmake config:
>
> if(PYARROW_USE_TENSORFLOW)
>   # TensorFlow uses the old GLIBCXX ABI, so we have to use it too
>   set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_USE_CXX11_ABI=0")
> endif()
>
> This was overriding my attempts to add c++11 to the pyarrow build (which
> wasn't a good idea, apparently).  Still working on the build per your
> suggestion. The gandiva test executable build chokes if c++11 isn't
> available, so I turned off the test build.
>
> On Mon, Oct 10, 2022 at 3:47 PM Joseph Porter <jo...@falkonry.com>
> wrote:
>
>> That makes sense. I'll give it a try - thanks!
>>
>> On Mon, Oct 10, 2022 at 2:34 PM Antoine Pitrou <an...@python.org>
>> wrote:
>>
>>>
>>>
>>> Le 10/10/2022 à 21:31, Joseph Porter a écrit :
>>> > No go. I still get the B5cxx11 extension on the symbols in the compiled
>>> > libarrow library.
>>>
>>> "-D_GLIBCXX_USE_CXX11_ABI=0" is a C++ compiler flag, not a CMake flag.
>>>
>>> One possibility is to pass instead
>>> "-DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0" to CMake.
>>>
>>>
>>> >
>>> > Tried:
>>> > /workspace/arrow/pyarrow-dev/bin/cmake
>>> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
>>> > -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release
>>> -DARROW_DATASET=ON
>>> > -DARROW_WITH_BZ2=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON
>>> > -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_BROTLI=ON
>>> > -DARROW_PARQUET=ON -DPARQUET_REQUIRE_ENCRYPTION=ON -DARROW_PYTHON=ON
>>> > -DARROW_BUILD_TESTS=ON -DARROW_CUDA=ON -DARROW_FLIGHT=ON
>>> -DARROW_GANDIVA=ON
>>> > -DARROW_PLASMA=ON -DARROW_S3=ON -DARROW_TENSORFLOW=ON -DARROW_CSV=ON
>>> > -DARROW_JSON=ON -DARROW_WITH_RE2=ON -DARROW_IPC=ON
>>> > -DARROW_DEPENDENCY_SOURCE=BUNDLED -D_GLIBCXX_USE_CXX11_ABI=0 ..
>>> >
>>> > For my next attempt, I modified CMakeLists.txt (in arrow/cpp) to
>>> explicitly
>>> > remove any mention of -std=c++11 in the CXXFLAGS.  I'll let you know if
>>> > that works.  Any pointers on better ways to troubleshoot this would
>>> also be
>>> > helpful.  I'm not super-conversant with CMake.
>>> >
>>> > # Remove --std=c++11 to avoid errors from C compilers
>>> > string(REPLACE "-std=c++11" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
>>> >
>>> > # Add C++-only flags, like -std=c++11
>>> > set(CMAKE_CXX_FLAGS "${CXX_ONLY_FLAGS} ${CMAKE_CXX_FLAGS}")
>>> > string(REPLACE "-std=c++11" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
>>> >
>>> > On Mon, Oct 10, 2022 at 1:29 PM Antoine Pitrou <an...@python.org>
>>> wrote:
>>> >
>>> >>
>>> >> Then instead pass "-D_GLIBCXX_USE_CXX11_ABI=0" when building the C++
>>> >> libraries?
>>> >>
>>> >>
>>> >> Le 10/10/2022 à 20:20, Joseph Porter a écrit :
>>> >>> Hi Antoine,
>>> >>>
>>> >>> Here's what I did:
>>> >>> export PYARROW_CXXFLAGS="-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1"
>>> >>>
>>> >>> Here's what I got:
>>> >>> ImportError:
>>> >>> /workspace/arrow/pyarrow-test/lib/python3.8/site-packages/pyarrow/
>>> >>> lib.cpython-38-x86_64-linux-gnu.so: undefined symbol:
>>> >>> _ZNK5arrow8DataType18ComputeFingerprintEv
>>> >>>
>>> >>> It looks like the symbol mismatch exists already in the libraries
>>> that
>>> >> were
>>> >>> created by the C++ build step, which is why I tried to add the c++11
>>> >>> directives to the CMakeLists.txt in the python module.
>>> >>>
>>> >>> -Joe
>>> >>>
>>> >>> On Mon, Oct 10, 2022 at 12:37 PM Antoine Pitrou <an...@python.org>
>>> >> wrote:
>>> >>>
>>> >>>>
>>> >>>> Le 10/10/2022 à 19:27, Joseph Porter a écrit :
>>> >>>>>
>>> >>>>> I've tried building with explicit flags to encourage the libraries
>>> to
>>> >>>>> include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't
>>> seem
>>> >>>> to
>>> >>>>> impact this issue:
>>> >>>>>
>>> >>>>> set (CMAKE_CXX_STANDARD 11)
>>> >>>>> set (CMAKE_CXX_STANDARD_REQUIRED ON)
>>> >>>>> set (CMAKE_CXX_EXTENSIONS OFF)
>>> >>>>>
>>> >>>>> I also added
>>> >>>>> export PYARROW_CXXFLAGS="-std=c++11"
>>> >>>>> for the wheel build of pyarrow (no effect).
>>> >>>>
>>> >>>> Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?
>>> >>>>
>>> >>>
>>> >>
>>> >
>>>
>>

Re: Build Frustrations

Posted by Joseph Porter <jo...@falkonry.com.INVALID>.
I think I found a culprit in the pyarrow cmake config:

if(PYARROW_USE_TENSORFLOW)
  # TensorFlow uses the old GLIBCXX ABI, so we have to use it too
  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_USE_CXX11_ABI=0")
endif()

This was overriding my attempts to add c++11 to the pyarrow build (which
wasn't a good idea, apparently).  Still working on the build per your
suggestion. The gandiva test executable build chokes if c++11 isn't
available, so I turned off the test build.

On Mon, Oct 10, 2022 at 3:47 PM Joseph Porter <jo...@falkonry.com>
wrote:

> That makes sense. I'll give it a try - thanks!
>
> On Mon, Oct 10, 2022 at 2:34 PM Antoine Pitrou <an...@python.org> wrote:
>
>>
>>
>> Le 10/10/2022 à 21:31, Joseph Porter a écrit :
>> > No go. I still get the B5cxx11 extension on the symbols in the compiled
>> > libarrow library.
>>
>> "-D_GLIBCXX_USE_CXX11_ABI=0" is a C++ compiler flag, not a CMake flag.
>>
>> One possibility is to pass instead
>> "-DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0" to CMake.
>>
>>
>> >
>> > Tried:
>> > /workspace/arrow/pyarrow-dev/bin/cmake
>> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
>> > -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DARROW_DATASET=ON
>> > -DARROW_WITH_BZ2=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON
>> > -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_BROTLI=ON
>> > -DARROW_PARQUET=ON -DPARQUET_REQUIRE_ENCRYPTION=ON -DARROW_PYTHON=ON
>> > -DARROW_BUILD_TESTS=ON -DARROW_CUDA=ON -DARROW_FLIGHT=ON
>> -DARROW_GANDIVA=ON
>> > -DARROW_PLASMA=ON -DARROW_S3=ON -DARROW_TENSORFLOW=ON -DARROW_CSV=ON
>> > -DARROW_JSON=ON -DARROW_WITH_RE2=ON -DARROW_IPC=ON
>> > -DARROW_DEPENDENCY_SOURCE=BUNDLED -D_GLIBCXX_USE_CXX11_ABI=0 ..
>> >
>> > For my next attempt, I modified CMakeLists.txt (in arrow/cpp) to
>> explicitly
>> > remove any mention of -std=c++11 in the CXXFLAGS.  I'll let you know if
>> > that works.  Any pointers on better ways to troubleshoot this would
>> also be
>> > helpful.  I'm not super-conversant with CMake.
>> >
>> > # Remove --std=c++11 to avoid errors from C compilers
>> > string(REPLACE "-std=c++11" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
>> >
>> > # Add C++-only flags, like -std=c++11
>> > set(CMAKE_CXX_FLAGS "${CXX_ONLY_FLAGS} ${CMAKE_CXX_FLAGS}")
>> > string(REPLACE "-std=c++11" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
>> >
>> > On Mon, Oct 10, 2022 at 1:29 PM Antoine Pitrou <an...@python.org>
>> wrote:
>> >
>> >>
>> >> Then instead pass "-D_GLIBCXX_USE_CXX11_ABI=0" when building the C++
>> >> libraries?
>> >>
>> >>
>> >> Le 10/10/2022 à 20:20, Joseph Porter a écrit :
>> >>> Hi Antoine,
>> >>>
>> >>> Here's what I did:
>> >>> export PYARROW_CXXFLAGS="-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1"
>> >>>
>> >>> Here's what I got:
>> >>> ImportError:
>> >>> /workspace/arrow/pyarrow-test/lib/python3.8/site-packages/pyarrow/
>> >>> lib.cpython-38-x86_64-linux-gnu.so: undefined symbol:
>> >>> _ZNK5arrow8DataType18ComputeFingerprintEv
>> >>>
>> >>> It looks like the symbol mismatch exists already in the libraries that
>> >> were
>> >>> created by the C++ build step, which is why I tried to add the c++11
>> >>> directives to the CMakeLists.txt in the python module.
>> >>>
>> >>> -Joe
>> >>>
>> >>> On Mon, Oct 10, 2022 at 12:37 PM Antoine Pitrou <an...@python.org>
>> >> wrote:
>> >>>
>> >>>>
>> >>>> Le 10/10/2022 à 19:27, Joseph Porter a écrit :
>> >>>>>
>> >>>>> I've tried building with explicit flags to encourage the libraries
>> to
>> >>>>> include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't
>> seem
>> >>>> to
>> >>>>> impact this issue:
>> >>>>>
>> >>>>> set (CMAKE_CXX_STANDARD 11)
>> >>>>> set (CMAKE_CXX_STANDARD_REQUIRED ON)
>> >>>>> set (CMAKE_CXX_EXTENSIONS OFF)
>> >>>>>
>> >>>>> I also added
>> >>>>> export PYARROW_CXXFLAGS="-std=c++11"
>> >>>>> for the wheel build of pyarrow (no effect).
>> >>>>
>> >>>> Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?
>> >>>>
>> >>>
>> >>
>> >
>>
>

Re: Build Frustrations

Posted by Joseph Porter <jo...@falkonry.com.INVALID>.
That makes sense. I'll give it a try - thanks!

On Mon, Oct 10, 2022 at 2:34 PM Antoine Pitrou <an...@python.org> wrote:

>
>
> Le 10/10/2022 à 21:31, Joseph Porter a écrit :
> > No go. I still get the B5cxx11 extension on the symbols in the compiled
> > libarrow library.
>
> "-D_GLIBCXX_USE_CXX11_ABI=0" is a C++ compiler flag, not a CMake flag.
>
> One possibility is to pass instead
> "-DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0" to CMake.
>
>
> >
> > Tried:
> > /workspace/arrow/pyarrow-dev/bin/cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
> > -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DARROW_DATASET=ON
> > -DARROW_WITH_BZ2=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON
> > -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_BROTLI=ON
> > -DARROW_PARQUET=ON -DPARQUET_REQUIRE_ENCRYPTION=ON -DARROW_PYTHON=ON
> > -DARROW_BUILD_TESTS=ON -DARROW_CUDA=ON -DARROW_FLIGHT=ON
> -DARROW_GANDIVA=ON
> > -DARROW_PLASMA=ON -DARROW_S3=ON -DARROW_TENSORFLOW=ON -DARROW_CSV=ON
> > -DARROW_JSON=ON -DARROW_WITH_RE2=ON -DARROW_IPC=ON
> > -DARROW_DEPENDENCY_SOURCE=BUNDLED -D_GLIBCXX_USE_CXX11_ABI=0 ..
> >
> > For my next attempt, I modified CMakeLists.txt (in arrow/cpp) to
> explicitly
> > remove any mention of -std=c++11 in the CXXFLAGS.  I'll let you know if
> > that works.  Any pointers on better ways to troubleshoot this would also
> be
> > helpful.  I'm not super-conversant with CMake.
> >
> > # Remove --std=c++11 to avoid errors from C compilers
> > string(REPLACE "-std=c++11" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
> >
> > # Add C++-only flags, like -std=c++11
> > set(CMAKE_CXX_FLAGS "${CXX_ONLY_FLAGS} ${CMAKE_CXX_FLAGS}")
> > string(REPLACE "-std=c++11" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
> >
> > On Mon, Oct 10, 2022 at 1:29 PM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> Then instead pass "-D_GLIBCXX_USE_CXX11_ABI=0" when building the C++
> >> libraries?
> >>
> >>
> >> Le 10/10/2022 à 20:20, Joseph Porter a écrit :
> >>> Hi Antoine,
> >>>
> >>> Here's what I did:
> >>> export PYARROW_CXXFLAGS="-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1"
> >>>
> >>> Here's what I got:
> >>> ImportError:
> >>> /workspace/arrow/pyarrow-test/lib/python3.8/site-packages/pyarrow/
> >>> lib.cpython-38-x86_64-linux-gnu.so: undefined symbol:
> >>> _ZNK5arrow8DataType18ComputeFingerprintEv
> >>>
> >>> It looks like the symbol mismatch exists already in the libraries that
> >> were
> >>> created by the C++ build step, which is why I tried to add the c++11
> >>> directives to the CMakeLists.txt in the python module.
> >>>
> >>> -Joe
> >>>
> >>> On Mon, Oct 10, 2022 at 12:37 PM Antoine Pitrou <an...@python.org>
> >> wrote:
> >>>
> >>>>
> >>>> Le 10/10/2022 à 19:27, Joseph Porter a écrit :
> >>>>>
> >>>>> I've tried building with explicit flags to encourage the libraries to
> >>>>> include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't
> seem
> >>>> to
> >>>>> impact this issue:
> >>>>>
> >>>>> set (CMAKE_CXX_STANDARD 11)
> >>>>> set (CMAKE_CXX_STANDARD_REQUIRED ON)
> >>>>> set (CMAKE_CXX_EXTENSIONS OFF)
> >>>>>
> >>>>> I also added
> >>>>> export PYARROW_CXXFLAGS="-std=c++11"
> >>>>> for the wheel build of pyarrow (no effect).
> >>>>
> >>>> Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?
> >>>>
> >>>
> >>
> >
>

Re: Build Frustrations

Posted by Antoine Pitrou <an...@python.org>.

Le 10/10/2022 à 21:31, Joseph Porter a écrit :
> No go. I still get the B5cxx11 extension on the symbols in the compiled
> libarrow library.

"-D_GLIBCXX_USE_CXX11_ABI=0" is a C++ compiler flag, not a CMake flag.

One possibility is to pass instead
"-DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0" to CMake.


> 
> Tried:
> /workspace/arrow/pyarrow-dev/bin/cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
> -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DARROW_DATASET=ON
> -DARROW_WITH_BZ2=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON
> -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_BROTLI=ON
> -DARROW_PARQUET=ON -DPARQUET_REQUIRE_ENCRYPTION=ON -DARROW_PYTHON=ON
> -DARROW_BUILD_TESTS=ON -DARROW_CUDA=ON -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON
> -DARROW_PLASMA=ON -DARROW_S3=ON -DARROW_TENSORFLOW=ON -DARROW_CSV=ON
> -DARROW_JSON=ON -DARROW_WITH_RE2=ON -DARROW_IPC=ON
> -DARROW_DEPENDENCY_SOURCE=BUNDLED -D_GLIBCXX_USE_CXX11_ABI=0 ..
> 
> For my next attempt, I modified CMakeLists.txt (in arrow/cpp) to explicitly
> remove any mention of -std=c++11 in the CXXFLAGS.  I'll let you know if
> that works.  Any pointers on better ways to troubleshoot this would also be
> helpful.  I'm not super-conversant with CMake.
> 
> # Remove --std=c++11 to avoid errors from C compilers
> string(REPLACE "-std=c++11" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
> 
> # Add C++-only flags, like -std=c++11
> set(CMAKE_CXX_FLAGS "${CXX_ONLY_FLAGS} ${CMAKE_CXX_FLAGS}")
> string(REPLACE "-std=c++11" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
> 
> On Mon, Oct 10, 2022 at 1:29 PM Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> Then instead pass "-D_GLIBCXX_USE_CXX11_ABI=0" when building the C++
>> libraries?
>>
>>
>> Le 10/10/2022 à 20:20, Joseph Porter a écrit :
>>> Hi Antoine,
>>>
>>> Here's what I did:
>>> export PYARROW_CXXFLAGS="-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1"
>>>
>>> Here's what I got:
>>> ImportError:
>>> /workspace/arrow/pyarrow-test/lib/python3.8/site-packages/pyarrow/
>>> lib.cpython-38-x86_64-linux-gnu.so: undefined symbol:
>>> _ZNK5arrow8DataType18ComputeFingerprintEv
>>>
>>> It looks like the symbol mismatch exists already in the libraries that
>> were
>>> created by the C++ build step, which is why I tried to add the c++11
>>> directives to the CMakeLists.txt in the python module.
>>>
>>> -Joe
>>>
>>> On Mon, Oct 10, 2022 at 12:37 PM Antoine Pitrou <an...@python.org>
>> wrote:
>>>
>>>>
>>>> Le 10/10/2022 à 19:27, Joseph Porter a écrit :
>>>>>
>>>>> I've tried building with explicit flags to encourage the libraries to
>>>>> include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't seem
>>>> to
>>>>> impact this issue:
>>>>>
>>>>> set (CMAKE_CXX_STANDARD 11)
>>>>> set (CMAKE_CXX_STANDARD_REQUIRED ON)
>>>>> set (CMAKE_CXX_EXTENSIONS OFF)
>>>>>
>>>>> I also added
>>>>> export PYARROW_CXXFLAGS="-std=c++11"
>>>>> for the wheel build of pyarrow (no effect).
>>>>
>>>> Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?
>>>>
>>>
>>
> 

Re: Build Frustrations

Posted by Joseph Porter <jo...@falkonry.com.INVALID>.
No go. I still get the B5cxx11 extension on the symbols in the compiled
libarrow library.

Tried:
/workspace/arrow/pyarrow-dev/bin/cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
-DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DARROW_DATASET=ON
-DARROW_WITH_BZ2=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON
-DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_BROTLI=ON
-DARROW_PARQUET=ON -DPARQUET_REQUIRE_ENCRYPTION=ON -DARROW_PYTHON=ON
-DARROW_BUILD_TESTS=ON -DARROW_CUDA=ON -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON
-DARROW_PLASMA=ON -DARROW_S3=ON -DARROW_TENSORFLOW=ON -DARROW_CSV=ON
-DARROW_JSON=ON -DARROW_WITH_RE2=ON -DARROW_IPC=ON
-DARROW_DEPENDENCY_SOURCE=BUNDLED -D_GLIBCXX_USE_CXX11_ABI=0 ..

For my next attempt, I modified CMakeLists.txt (in arrow/cpp) to explicitly
remove any mention of -std=c++11 in the CXXFLAGS.  I'll let you know if
that works.  Any pointers on better ways to troubleshoot this would also be
helpful.  I'm not super-conversant with CMake.

# Remove --std=c++11 to avoid errors from C compilers
string(REPLACE "-std=c++11" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})

# Add C++-only flags, like -std=c++11
set(CMAKE_CXX_FLAGS "${CXX_ONLY_FLAGS} ${CMAKE_CXX_FLAGS}")
string(REPLACE "-std=c++11" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})

On Mon, Oct 10, 2022 at 1:29 PM Antoine Pitrou <an...@python.org> wrote:

>
> Then instead pass "-D_GLIBCXX_USE_CXX11_ABI=0" when building the C++
> libraries?
>
>
> Le 10/10/2022 à 20:20, Joseph Porter a écrit :
> > Hi Antoine,
> >
> > Here's what I did:
> > export PYARROW_CXXFLAGS="-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1"
> >
> > Here's what I got:
> > ImportError:
> > /workspace/arrow/pyarrow-test/lib/python3.8/site-packages/pyarrow/
> > lib.cpython-38-x86_64-linux-gnu.so: undefined symbol:
> > _ZNK5arrow8DataType18ComputeFingerprintEv
> >
> > It looks like the symbol mismatch exists already in the libraries that
> were
> > created by the C++ build step, which is why I tried to add the c++11
> > directives to the CMakeLists.txt in the python module.
> >
> > -Joe
> >
> > On Mon, Oct 10, 2022 at 12:37 PM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> Le 10/10/2022 à 19:27, Joseph Porter a écrit :
> >>>
> >>> I've tried building with explicit flags to encourage the libraries to
> >>> include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't seem
> >> to
> >>> impact this issue:
> >>>
> >>> set (CMAKE_CXX_STANDARD 11)
> >>> set (CMAKE_CXX_STANDARD_REQUIRED ON)
> >>> set (CMAKE_CXX_EXTENSIONS OFF)
> >>>
> >>> I also added
> >>> export PYARROW_CXXFLAGS="-std=c++11"
> >>> for the wheel build of pyarrow (no effect).
> >>
> >> Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?
> >>
> >
>

Re: Build Frustrations

Posted by Antoine Pitrou <an...@python.org>.
Then instead pass "-D_GLIBCXX_USE_CXX11_ABI=0" when building the C++ 
libraries?


Le 10/10/2022 à 20:20, Joseph Porter a écrit :
> Hi Antoine,
> 
> Here's what I did:
> export PYARROW_CXXFLAGS="-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1"
> 
> Here's what I got:
> ImportError:
> /workspace/arrow/pyarrow-test/lib/python3.8/site-packages/pyarrow/
> lib.cpython-38-x86_64-linux-gnu.so: undefined symbol:
> _ZNK5arrow8DataType18ComputeFingerprintEv
> 
> It looks like the symbol mismatch exists already in the libraries that were
> created by the C++ build step, which is why I tried to add the c++11
> directives to the CMakeLists.txt in the python module.
> 
> -Joe
> 
> On Mon, Oct 10, 2022 at 12:37 PM Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> Le 10/10/2022 à 19:27, Joseph Porter a écrit :
>>>
>>> I've tried building with explicit flags to encourage the libraries to
>>> include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't seem
>> to
>>> impact this issue:
>>>
>>> set (CMAKE_CXX_STANDARD 11)
>>> set (CMAKE_CXX_STANDARD_REQUIRED ON)
>>> set (CMAKE_CXX_EXTENSIONS OFF)
>>>
>>> I also added
>>> export PYARROW_CXXFLAGS="-std=c++11"
>>> for the wheel build of pyarrow (no effect).
>>
>> Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?
>>
> 

Re: Build Frustrations

Posted by Joseph Porter <jo...@falkonry.com.INVALID>.
Hi Antoine,

Here's what I did:
export PYARROW_CXXFLAGS="-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1"

Here's what I got:
ImportError:
/workspace/arrow/pyarrow-test/lib/python3.8/site-packages/pyarrow/
lib.cpython-38-x86_64-linux-gnu.so: undefined symbol:
_ZNK5arrow8DataType18ComputeFingerprintEv

It looks like the symbol mismatch exists already in the libraries that were
created by the C++ build step, which is why I tried to add the c++11
directives to the CMakeLists.txt in the python module.

-Joe

On Mon, Oct 10, 2022 at 12:37 PM Antoine Pitrou <an...@python.org> wrote:

>
> Le 10/10/2022 à 19:27, Joseph Porter a écrit :
> >
> > I've tried building with explicit flags to encourage the libraries to
> > include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't seem
> to
> > impact this issue:
> >
> > set (CMAKE_CXX_STANDARD 11)
> > set (CMAKE_CXX_STANDARD_REQUIRED ON)
> > set (CMAKE_CXX_EXTENSIONS OFF)
> >
> > I also added
> > export PYARROW_CXXFLAGS="-std=c++11"
> > for the wheel build of pyarrow (no effect).
>
> Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?
>

Re: Build Frustrations

Posted by Antoine Pitrou <an...@python.org>.
Le 10/10/2022 à 19:27, Joseph Porter a écrit :
> 
> I've tried building with explicit flags to encourage the libraries to
> include the cxx11 symbol (in python/CMakeLists.txt).  That doesn't seem to
> impact this issue:
> 
> set (CMAKE_CXX_STANDARD 11)
> set (CMAKE_CXX_STANDARD_REQUIRED ON)
> set (CMAKE_CXX_EXTENSIONS OFF)
> 
> I also added
> export PYARROW_CXXFLAGS="-std=c++11"
> for the wheel build of pyarrow (no effect).

Can you try adding "-D_GLIBCXX_USE_CXX11_ABI=1" to those flags?