You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Brian Bowman <Br...@sas.com.INVALID> on 2020/04/02 16:56:06 UTC
CPP : arrow symbols.map issue
A new high-performance file system we are working with returns an error while writing a .parquet file. The following arrow symbol does not resolve properly and the error is masked.
libparquet.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev
> nm libarrow.so* | grep -i ZNK5arrow6Status8ToStringB5cxx11Ev
00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
One of our Linux dev/build experts tracked this down to an issue in arrow open source. He says the lowercase ‘t’ (text) code (… 7760 t _ZNK …) in the nm command output is incorrect and it should instead be an uppercase ‘T’.
He traced the problem to this file:
../cpp/src/arrow/symbols.map
Here’s an update with his fix. Lines 27-30 are new. Nothing else changes.
1 # Licensed to the Apache Software Foundation (ASF) under one
2 # or more contributor license agreements. See the NOTICE file
3 # distributed with this work for additional information
4 # regarding copyright ownership. The ASF licenses this file
5 # to you under the Apache License, Version 2.0 (the
6 # "License"); you may not use this file except in compliance
7 # with the License. You may obtain a copy of the License at
8 #
9 # http://www.apache.org/licenses/LICENSE-2.0
10 #
11 # Unless required by applicable law or agreed to in writing,
12 # software distributed under the License is distributed on an
13 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14 # KIND, either express or implied. See the License for the
15 # specific language governing permissions and limitations
16 # under the License.
17
18 {
19 global:
20 extern "C++" {
21 # The leading asterisk is required for symbols such as
22 # "typeinfo for arrow::SomeClass".
23 # Unfortunately this will also catch template specializations
24 # (from e.g. STL or Flatbuffers) involving Arrow types.
25 *arrow::*;
26 *arrow_vendored::*;
27 *ToString*;
28 *key*;
29 *str*;
30 *value*;
31 };
32 # Also export C-level helpers
33 arrow_*;
34 pyarrow_*;
35
36 # Symbols marked as 'local' are not exported by the DSO and thus may not
37 # be used by client applications. Everything except the above falls here.
38 # This ensures we hide symbols of static dependencies.
39 local:
40 *;
41
42 };
We have made these changes in our local clones the arrow open source repositories. I’m passing this along for the community’s review. Reply with a link and I’ll enter a jira ticket if needed.
-Brian
Re: CPP : arrow symbols.map issue
Posted by Brian Bowman <Br...@sas.com.INVALID>.
Have not tried to reproduce in containerized environment Wes.
-Brian
On 4/3/20, 11:42 AM, "Wes McKinney" <we...@gmail.com> wrote:
EXTERNAL
Are you able to reproduce the issue in a Dockerfile?
Because you are building with gcc 7.4, you need to ensure either that you
build everything with the gcc < 5 ABI otherwise (if you want the new gcc
ABI) ensure that the machine where you deploy has libstd++ for gcc 7. Using
Redhat's devtoolset toolchain also an option.
On Fri, Apr 3, 2020, 10:01 AM Brian Bowman <Br...@sas.com.invalid>
wrote:
> Antoine/Wes,
>
> Thanks for your assistance!
>
> Here is the relevant info. We suspect that our production build machines
> being at RHEL 6.7 is an issue.
>
> OS: RHEL 6.7
> Tools: gcc 7.4, bison-3.2, cmake-3.13.1, automake 1.16.1, autoconf 2.69,
> libtool 2.4.6, pkgcnf 1.1.0, texinfo 6.6, help2man 1.47.11, ld
> 2.20.51.0.2-5.43.el6
>
> Best,
>
> -Brian
>
> On 4/2/20, 1:22 PM, "Wes McKinney" <we...@gmail.com> wrote:
>
> EXTERNAL
>
> On Thu, Apr 2, 2020 at 12:06 PM Antoine Pitrou <so...@pitrou.net>
> wrote:
> >
> >
> > Hi,
> >
> > On Thu, 2 Apr 2020 16:56:06 +0000
> > Brian Bowman <Br...@sas.com.INVALID> wrote:
> > > A new high-performance file system we are working with returns an
> error while writing a .parquet file. The following arrow symbol does not
> resolve properly and the error is masked.
> > >
> > > libparquet.so: undefined symbol:
> _ZNK5arrow6Status8ToStringB5cxx11Ev
> > >
> > > > nm libarrow.so* | grep -i ZNK5arrow6Status8ToStringB5cxx11Ev
> > > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
> > > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
> >
> > For clarity, you should use `nm --demangle`. This will give you the
> > actual C++ symbol, i.e. "arrow::Status::ToString[abi:cxx11]() const".
> >
> > > One of our Linux dev/build experts tracked this down to an issue
> in arrow open source. He says the lowercase ‘t’ (text) code (… 7760 t _ZNK
> …) in the nm command output is incorrect and it should instead be an
> uppercase ‘T’.
> >
> > I have the right output here:
> >
> > $ nm --demangle --defined-only --dynamic .../libarrow.so | \
> > grep Status::ToString
> > 00000000012f1ff0 T arrow::Status::ToString[abi:cxx11]() const
> >
> > Which toolchain (linker etc.) are you using?
>
> My guess is also that you have a mixed-gcc-toolchain problem. What
> compiler/linker (and gcc toolchain, if you built with Clang) was used
> to produce libparquet.so (or where did you obtain the package), and
> which toolchain are you using to build and link your application?
>
> > Regards
> >
> > Antoine.
> >
> >
>
>
>
Re: CPP : arrow symbols.map issue
Posted by Wes McKinney <we...@gmail.com>.
Are you able to reproduce the issue in a Dockerfile?
Because you are building with gcc 7.4, you need to ensure either that you
build everything with the gcc < 5 ABI otherwise (if you want the new gcc
ABI) ensure that the machine where you deploy has libstd++ for gcc 7. Using
Redhat's devtoolset toolchain also an option.
On Fri, Apr 3, 2020, 10:01 AM Brian Bowman <Br...@sas.com.invalid>
wrote:
> Antoine/Wes,
>
> Thanks for your assistance!
>
> Here is the relevant info. We suspect that our production build machines
> being at RHEL 6.7 is an issue.
>
> OS: RHEL 6.7
> Tools: gcc 7.4, bison-3.2, cmake-3.13.1, automake 1.16.1, autoconf 2.69,
> libtool 2.4.6, pkgcnf 1.1.0, texinfo 6.6, help2man 1.47.11, ld
> 2.20.51.0.2-5.43.el6
>
> Best,
>
> -Brian
>
> On 4/2/20, 1:22 PM, "Wes McKinney" <we...@gmail.com> wrote:
>
> EXTERNAL
>
> On Thu, Apr 2, 2020 at 12:06 PM Antoine Pitrou <so...@pitrou.net>
> wrote:
> >
> >
> > Hi,
> >
> > On Thu, 2 Apr 2020 16:56:06 +0000
> > Brian Bowman <Br...@sas.com.INVALID> wrote:
> > > A new high-performance file system we are working with returns an
> error while writing a .parquet file. The following arrow symbol does not
> resolve properly and the error is masked.
> > >
> > > libparquet.so: undefined symbol:
> _ZNK5arrow6Status8ToStringB5cxx11Ev
> > >
> > > > nm libarrow.so* | grep -i ZNK5arrow6Status8ToStringB5cxx11Ev
> > > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
> > > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
> >
> > For clarity, you should use `nm --demangle`. This will give you the
> > actual C++ symbol, i.e. "arrow::Status::ToString[abi:cxx11]() const".
> >
> > > One of our Linux dev/build experts tracked this down to an issue
> in arrow open source. He says the lowercase ‘t’ (text) code (… 7760 t _ZNK
> …) in the nm command output is incorrect and it should instead be an
> uppercase ‘T’.
> >
> > I have the right output here:
> >
> > $ nm --demangle --defined-only --dynamic .../libarrow.so | \
> > grep Status::ToString
> > 00000000012f1ff0 T arrow::Status::ToString[abi:cxx11]() const
> >
> > Which toolchain (linker etc.) are you using?
>
> My guess is also that you have a mixed-gcc-toolchain problem. What
> compiler/linker (and gcc toolchain, if you built with Clang) was used
> to produce libparquet.so (or where did you obtain the package), and
> which toolchain are you using to build and link your application?
>
> > Regards
> >
> > Antoine.
> >
> >
>
>
>
Re: CPP : arrow symbols.map issue
Posted by Brian Bowman <Br...@sas.com.INVALID>.
Antoine/Wes,
Thanks for your assistance!
Here is the relevant info. We suspect that our production build machines being at RHEL 6.7 is an issue.
OS: RHEL 6.7
Tools: gcc 7.4, bison-3.2, cmake-3.13.1, automake 1.16.1, autoconf 2.69, libtool 2.4.6, pkgcnf 1.1.0, texinfo 6.6, help2man 1.47.11, ld 2.20.51.0.2-5.43.el6
Best,
-Brian
On 4/2/20, 1:22 PM, "Wes McKinney" <we...@gmail.com> wrote:
EXTERNAL
On Thu, Apr 2, 2020 at 12:06 PM Antoine Pitrou <so...@pitrou.net> wrote:
>
>
> Hi,
>
> On Thu, 2 Apr 2020 16:56:06 +0000
> Brian Bowman <Br...@sas.com.INVALID> wrote:
> > A new high-performance file system we are working with returns an error while writing a .parquet file. The following arrow symbol does not resolve properly and the error is masked.
> >
> > libparquet.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev
> >
> > > nm libarrow.so* | grep -i ZNK5arrow6Status8ToStringB5cxx11Ev
> > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
> > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
>
> For clarity, you should use `nm --demangle`. This will give you the
> actual C++ symbol, i.e. "arrow::Status::ToString[abi:cxx11]() const".
>
> > One of our Linux dev/build experts tracked this down to an issue in arrow open source. He says the lowercase ‘t’ (text) code (… 7760 t _ZNK …) in the nm command output is incorrect and it should instead be an uppercase ‘T’.
>
> I have the right output here:
>
> $ nm --demangle --defined-only --dynamic .../libarrow.so | \
> grep Status::ToString
> 00000000012f1ff0 T arrow::Status::ToString[abi:cxx11]() const
>
> Which toolchain (linker etc.) are you using?
My guess is also that you have a mixed-gcc-toolchain problem. What
compiler/linker (and gcc toolchain, if you built with Clang) was used
to produce libparquet.so (or where did you obtain the package), and
which toolchain are you using to build and link your application?
> Regards
>
> Antoine.
>
>
Re: CPP : arrow symbols.map issue
Posted by Wes McKinney <we...@gmail.com>.
On Thu, Apr 2, 2020 at 12:06 PM Antoine Pitrou <so...@pitrou.net> wrote:
>
>
> Hi,
>
> On Thu, 2 Apr 2020 16:56:06 +0000
> Brian Bowman <Br...@sas.com.INVALID> wrote:
> > A new high-performance file system we are working with returns an error while writing a .parquet file. The following arrow symbol does not resolve properly and the error is masked.
> >
> > libparquet.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev
> >
> > > nm libarrow.so* | grep -i ZNK5arrow6Status8ToStringB5cxx11Ev
> > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
> > 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
>
> For clarity, you should use `nm --demangle`. This will give you the
> actual C++ symbol, i.e. "arrow::Status::ToString[abi:cxx11]() const".
>
> > One of our Linux dev/build experts tracked this down to an issue in arrow open source. He says the lowercase ‘t’ (text) code (… 7760 t _ZNK …) in the nm command output is incorrect and it should instead be an uppercase ‘T’.
>
> I have the right output here:
>
> $ nm --demangle --defined-only --dynamic .../libarrow.so | \
> grep Status::ToString
> 00000000012f1ff0 T arrow::Status::ToString[abi:cxx11]() const
>
> Which toolchain (linker etc.) are you using?
My guess is also that you have a mixed-gcc-toolchain problem. What
compiler/linker (and gcc toolchain, if you built with Clang) was used
to produce libparquet.so (or where did you obtain the package), and
which toolchain are you using to build and link your application?
> Regards
>
> Antoine.
>
>
Re: CPP : arrow symbols.map issue
Posted by Antoine Pitrou <so...@pitrou.net>.
Hi,
On Thu, 2 Apr 2020 16:56:06 +0000
Brian Bowman <Br...@sas.com.INVALID> wrote:
> A new high-performance file system we are working with returns an error while writing a .parquet file. The following arrow symbol does not resolve properly and the error is masked.
>
> libparquet.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev
>
> > nm libarrow.so* | grep -i ZNK5arrow6Status8ToStringB5cxx11Ev
> 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
> 00000000002b7760 t _ZNK5arrow6Status8ToStringB5cxx11Ev
For clarity, you should use `nm --demangle`. This will give you the
actual C++ symbol, i.e. "arrow::Status::ToString[abi:cxx11]() const".
> One of our Linux dev/build experts tracked this down to an issue in arrow open source. He says the lowercase ‘t’ (text) code (… 7760 t _ZNK …) in the nm command output is incorrect and it should instead be an uppercase ‘T’.
I have the right output here:
$ nm --demangle --defined-only --dynamic .../libarrow.so | \
grep Status::ToString
00000000012f1ff0 T arrow::Status::ToString[abi:cxx11]() const
Which toolchain (linker etc.) are you using?
Regards
Antoine.