You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Dimitris Tsirogiannis <dt...@cloudera.com> on 2016/08/17 17:10:32 UTC

Error linking impalad with -build_shared_libs option

Hey guys,

When I build from the latest master using the -build_shared_libs option I
get linking errors (see [1]). Building with static works fine. Any ideas?

Thanks
Dimitris


[1]
Linking CXX executable ../../build/debug/service/impalad
../../build/debug/util/libUtil.so: error: undefined reference to
'BZ2_bzBuffToBuffCompress'
../../build/debug/util/libUtil.so: error: undefined reference to
'snappy::MaxCompressedLength(unsigned long)'
../../build/debug/util/libUtil.so: error: undefined reference to
'snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)'
../../build/debug/util/libUtil.so: error: undefined reference to
'LZ4_compress'
../../build/debug/util/libUtil.so: error: undefined reference to
'BZ2_bzDecompressEnd'
../../build/debug/util/libUtil.so: error: undefined reference to
'BZ2_bzDecompressInit'
../../build/debug/util/libUtil.so: error: undefined reference to
'BZ2_bzBuffToBuffDecompress'
../../build/debug/util/libUtil.so: error: undefined reference to
'BZ2_bzDecompress'
../../build/debug/util/libUtil.so: error: undefined reference to
'snappy::GetUncompressedLength(char const*, unsigned long, unsigned long*)'
../../build/debug/util/libUtil.so: error: undefined reference to
'snappy::RawUncompress(char const*, unsigned long, char*)'
../../build/debug/util/libUtil.so: error: undefined reference to
'LZ4_uncompress'
../../build/debug/util/libUtil.so: error: undefined reference to 'my_strlen'
../../build/debug/util/libUtil.so: error: undefined reference to
'google_breakpad::ExceptionHandler::WriteMinidump(std::string const&, bool
(*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
../../build/debug/util/libUtil.so: error: undefined reference to
'google_breakpad::ExceptionHandler::ExceptionHandler(google_breakpad::MinidumpDescriptor
const&, bool (*)(void*), bool (*)(google_breakpad::MinidumpDescriptor
const&, void*, bool), void*, bool, int)'
../../build/debug/util/libUtil.so: error: undefined reference to
'ProfilerStart'
../../build/debug/util/libUtil.so: error: undefined reference to
'ProfilerStop'
../../build/debug/util/libUtil.so: error: undefined reference to
'boost::basic_regex<char, boost::regex_traits<char,
boost::cpp_regex_traits<char> > >::do_assign(char const*, char const*,
unsigned int)'
../../build/debug/util/libUtil.so: error: undefined reference to
'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char const*,
std::string>,
std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*,
std::string> > >, boost::regex_traits<char, boost::cpp_regex_traits<char> >
>::find()'
../../build/debug/util/libUtil.so: error: undefined reference to
'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char const*,
std::string>,
std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*,
std::string> > >, boost::regex_traits<char, boost::cpp_regex_traits<char> >
>::construct_init(boost::basic_regex<char, boost::regex_traits<char,
boost::cpp_regex_traits<char> > > const&,
boost::regex_constants::_match_flags)'
/opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(os-ip.o):os-ip.c:function
ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or
`strerror_r' instead
/opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(os-ip.o):os-ip.c:function
ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or
`strerror_r' instead
collect2: error: ld returned 1 exit status
be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe for target
'be/build/debug/service/impalad' failed
make[3]: *** [be/build/debug/service/impalad] Error 1
CMakeFiles/Makefile2:3682: recipe for target
'be/src/service/CMakeFiles/impalad.dir/all' failed
make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
CMakeFiles/Makefile2:3694: recipe for target
'be/src/service/CMakeFiles/impalad.dir/rule' failed
make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2

Re: Error linking impalad with -build_shared_libs option

Posted by Yonghyun Hwang <yo...@cloudera.com>.
Hello Martin,

Thank you for the fix. Just one curiosity. Do you know why the build issue
does not happen with the cmake 2.8.12.2 (that seems to be the system
default cmake in ubuntu 14.04)?

Thank you,
Yonghyun

On Thu, Aug 25, 2016 at 10:00 AM, Tim Armstrong <ta...@cloudera.com>
wrote:

> I merged the fix so hopefully this should now work for everyone - thanks
> Martin for helping us out here.
>
> On Thu, Aug 25, 2016 at 9:03 AM, Martin Grund (Das Grundprinzip.de) <
> grundprinzip@gmail.com> wrote:
>
> > What happened is that the Util library was linked to the error-utils-test
> > (which it is by default anyways because of the way how ADD_BE_TEST
> works).
> > Now, what happened is that if you dynamically link the order of the
> > libraries plays a role. The additional target-link-library would move the
> > libUtil.so to the end of the library list and if the linker picks this up
> > some of the symbols declared earlier where already dropped.
> >
> > On Wed, Aug 24, 2016 at 11:00 AM Henry Robinson <he...@cloudera.com>
> > wrote:
> >
> > > Seems great to me - but do we know why this fixes the issue? Not that
> > that
> > > should prevent us from merging the patch.
> > >
> > > On 24 August 2016 at 08:54, Tim Armstrong <ta...@cloudera.com>
> > wrote:
> > >
> > > > The fix works for me too, thanks Martin!
> > > >
> > > > I gave the patch a +1 but will upgrade to +2 if it seems everyone is
> > > happy
> > > > with it.
> > > >
> > > > Cheers,
> > > > Tim
> > > >
> > > > On Wed, Aug 24, 2016 at 8:15 AM, Yonghyun Hwang <
> yonghyun@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Hello Martin,
> > > > >
> > > > > Your fix works great! :) As for the build failure, what I've found
> is
> > > > that
> > > > > cmake 3.2.3 causes a build failure w/o your fix while cmake 2.8.12
> > > works
> > > > > good.
> > > > >
> > > > > After the commit for "IMPALA-3924: Ubuntu16 support", cmake 3.2.3
> got
> > > > > deployed under "toolchain", which reveals this build issue.
> > > > >
> > > > > So, any one using the newer cmake would require your fix to
> properly
> > > > build
> > > > > impala w/ -so.
> > > > >
> > > > > Thank you,
> > > > > Yonghyun
> > > > >
> > > > > On Tue, Aug 23, 2016 at 11:40 PM, Daniel Hecht <
> dhecht@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Martin,
> > > > > >
> > > > > > Nice to hear from you.
> > > https://issues.cloudera.org/browse/IMPALA-3979
> > > > > >
> > > > > > Thanks,
> > > > > > Dan
> > > > > >
> > > > > > On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
> > > > > > grundprinzip@gmail.com> wrote:
> > > > > >
> > > > > > > I've seen this in the past and propose a patch to fix it:
> > > > > > > https://gerrit.cloudera.org/#/c/4108/
> > > > > > >
> > > > > > > Let me know if you have JIRA to track this to update the commit
> > > > > message.
> > > > > > >
> > > > > > > Martin
> > > > > > >
> > > > > > > On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <
> > > > laszlo.gaal@cloudera.com
> > > > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > link.txt is generated by cmake based on the contents of
> > > > > CMakeLists.txt
> > > > > > > and
> > > > > > > > the project/environment settings. Most of the link-time
> > > > dependencies
> > > > > > seem
> > > > > > > > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on
> > > > settings
> > > > > > > > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > > > > > > > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look
> more
> > > > into
> > > > > it
> > > > > > > > tomorrow.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > >     - LaszloG
> > > > > > > >
> > > > > > > > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <
> > > > > yonghyun@cloudera.com
> > > > > > > <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I observed the same linking error and digged this into
> > further.
> > > > The
> > > > > > > below
> > > > > > > > > is what I've found.
> > > > > > > > >
> > > > > > > > > *1. how to reproduce*
> > > > > > > > >
> > > > > > > > > # assuming that impala-config.sh is sourced
> > > > > > > > > $ cd ${IMPALA_HOME}
> > > > > > > > > *$ make -j8 impalad*
> > > > > > > > > ...
> > > > > > > > > [100%] Built target CodeGen
> > > > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > > > ...
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > > `google_breakpad::ExceptionHandler::
> > WriteMinidump(std::string
> > > > > > const&,
> > > > > > > > bool
> > > > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*,
> > bool),
> > > > > > void*)'
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > > `snappy::MaxCompressedLength(unsigned long)'
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > `my_strlen'
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > `LZ4_compress'
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > > `BZ2_bzDecompress'
> > > > > > > > > ...
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *$ cd ${IMPALA_HOME}/be/src/service*
> > > > > > > > > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > > > > > > > > *cmake_link_script CMakeFiles/impalad.dir/link.txt
> > --verbose=*
> > > > > > > > > ...
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > > ...
> > > > > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > `LZ4_compress'*
> > > > > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > > `BZ2_bzDecompress'*
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > > *`snappy::RawCompress*(char const*,
> > > > > > > > > unsigned long, char*, unsigned long*)'
> > > > > > > > > ...
> > > > > > > > > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> > > > > > > > >::do_assign(char
> > > > > > > > > const*, char
> > > > > > > > > const*, unsigned int)'
> > > > > > > > > ...
> > > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > *`snappy::*
> > > > > > > > > RawUncompress(char
> > > > > > > > > const*, unsigned long, char*)'
> > > > > > > > >
> > > > > > > > > *2. cause of the failure*
> > > > > > > > >
> > > > > > > > > $ cd ${IMPALA_HOME}/be/src/service
> > > > > > > > > *$ cat CMakeFiles/impalad.dir/link.txt*
> > > > > > > > > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall
> > > > > > -Wno-sign-compare
> > > > > > > > > -Wno-unknown-pragmas -pthread -fno-strict-aliasing
> -std=c++14
> > > > > > > > > -Wno-deprecated ...
> > > > > > > > > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > > > > > > > > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > > > > > > > > ...
> > > > > > > > > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > > > > > > > > ...
> > > > > > > > > *../../build/debug/util/libUtil.so*
> > > > > > > > > -lpthread -ldl -lssl -lcrypto
> > > > > > > > >
> > > > > > > > > Apparently, libUtil.so __should__ show before libsnappy.a
> and
> > > > > > *regex.a.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Next step is to find out who creates
> > > CMakeFiles/impalad.dir/link.
> > > > > txt
> > > > > > > and
> > > > > > > > > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> > > > > > > > libboost_regex.a
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > > Yonghyun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <
> > > > > henry@cloudera.com
> > > > > > > <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Only that it's IMPALA-3979. Does anyone have any theories
> > > yet?
> > > > > > > > > >
> > > > > > > > > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > > > > > > > > dtsirogiannis@cloudera.com <javascript:;>> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey guys,
> > > > > > > > > > >
> > > > > > > > > > > When I build from the latest master using the
> > > > > -build_shared_libs
> > > > > > > > > option I
> > > > > > > > > > > get linking errors (see [1]). Building with static
> works
> > > > fine.
> > > > > > Any
> > > > > > > > > ideas?
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Dimitris
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > > Linking CXX executable ../../build/debug/service/
> impalad
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'BZ2_bzBuffToBuffCompress'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'snappy::RawCompress(char const*, unsigned long, char*,
> > > > > unsigned
> > > > > > > > > long*)'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'LZ4_compress'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'BZ2_bzDecompressEnd'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'BZ2_bzDecompressInit'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'BZ2_bzBuffToBuffDecompress'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'BZ2_bzDecompress'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'snappy::GetUncompressedLength(char const*, unsigned
> > long,
> > > > > > > unsigned
> > > > > > > > > > > long*)'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'snappy::RawUncompress(char const*, unsigned long,
> > char*)'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'LZ4_uncompress'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'my_strlen'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'google_breakpad::ExceptionHandler::
> > > > WriteMinidump(std::string
> > > > > > > const&,
> > > > > > > > > > bool
> > > > > > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*,
> > > > bool),
> > > > > > > void*)'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > >
> > > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > > > > > > > > breakpad::MinidumpDescriptor
> > > > > > > > > > > const&, bool (*)(void*), bool (*)(google_breakpad::
> > > > > > > MinidumpDescriptor
> > > > > > > > > > > const&, void*, bool), void*, bool, int)'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'ProfilerStart'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'ProfilerStop'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > > > > > > > > boost::cpp_regex_traits<char> > >::do_assign(char
> const*,
> > > > char
> > > > > > > > const*,
> > > > > > > > > > > unsigned int)'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > > > > iterator<char
> > > > > > > > > > const*,
> > > > > > > > > > > std::string>,
> > > > > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > > > > iterator<char
> > > > > > > > > > const*,
> > > > > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > > > > >::find()'
> > > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > > reference
> > > > > to
> > > > > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > > > > iterator<char
> > > > > > > > > > const*,
> > > > > > > > > > > std::string>,
> > > > > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > > > > iterator<char
> > > > > > > > > > const*,
> > > > > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > > > > >::construct_init(boost::basic_regex<char,
> > > > > > > boost::regex_traits<char,
> > > > > > > > > > > boost::cpp_regex_traits<char> > > const&,
> > > > > > > > > > > boost::regex_constants::_match_flags)'
> > > > > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use
> > > > > `strerror'
> > > > > > or
> > > > > > > > > > > `strerror_r' instead
> > > > > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > > > > ldap_int_poll: warning: `sys_errlist' is deprecated;
> use
> > > > > > `strerror'
> > > > > > > > or
> > > > > > > > > > > `strerror_r' instead
> > > > > > > > > > > collect2: error: ld returned 1 exit status
> > > > > > > > > > > be/src/service/CMakeFiles/impalad.dir/build.make:166:
> > > recipe
> > > > > for
> > > > > > > > > target
> > > > > > > > > > > 'be/build/debug/service/impalad' failed
> > > > > > > > > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > > > > > > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > > > > > > > > make[2]: *** [be/src/service/CMakeFiles/
> impalad.dir/all]
> > > > > Error 2
> > > > > > > > > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > > > > > > > > make[1]: *** [be/src/service/CMakeFiles/
> > impalad.dir/rule]
> > > > > Error
> > > > > > 2
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Henry Robinson
> > > > > > > > > > Software Engineer
> > > > > > > > > > Cloudera
> > > > > > > > > > 415-994-6679
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
>

Re: Error linking impalad with -build_shared_libs option

Posted by Tim Armstrong <ta...@cloudera.com>.
I merged the fix so hopefully this should now work for everyone - thanks
Martin for helping us out here.

On Thu, Aug 25, 2016 at 9:03 AM, Martin Grund (Das Grundprinzip.de) <
grundprinzip@gmail.com> wrote:

> What happened is that the Util library was linked to the error-utils-test
> (which it is by default anyways because of the way how ADD_BE_TEST works).
> Now, what happened is that if you dynamically link the order of the
> libraries plays a role. The additional target-link-library would move the
> libUtil.so to the end of the library list and if the linker picks this up
> some of the symbols declared earlier where already dropped.
>
> On Wed, Aug 24, 2016 at 11:00 AM Henry Robinson <he...@cloudera.com>
> wrote:
>
> > Seems great to me - but do we know why this fixes the issue? Not that
> that
> > should prevent us from merging the patch.
> >
> > On 24 August 2016 at 08:54, Tim Armstrong <ta...@cloudera.com>
> wrote:
> >
> > > The fix works for me too, thanks Martin!
> > >
> > > I gave the patch a +1 but will upgrade to +2 if it seems everyone is
> > happy
> > > with it.
> > >
> > > Cheers,
> > > Tim
> > >
> > > On Wed, Aug 24, 2016 at 8:15 AM, Yonghyun Hwang <yonghyun@cloudera.com
> >
> > > wrote:
> > >
> > > > Hello Martin,
> > > >
> > > > Your fix works great! :) As for the build failure, what I've found is
> > > that
> > > > cmake 3.2.3 causes a build failure w/o your fix while cmake 2.8.12
> > works
> > > > good.
> > > >
> > > > After the commit for "IMPALA-3924: Ubuntu16 support", cmake 3.2.3 got
> > > > deployed under "toolchain", which reveals this build issue.
> > > >
> > > > So, any one using the newer cmake would require your fix to properly
> > > build
> > > > impala w/ -so.
> > > >
> > > > Thank you,
> > > > Yonghyun
> > > >
> > > > On Tue, Aug 23, 2016 at 11:40 PM, Daniel Hecht <dh...@cloudera.com>
> > > > wrote:
> > > >
> > > > > Hi Martin,
> > > > >
> > > > > Nice to hear from you.
> > https://issues.cloudera.org/browse/IMPALA-3979
> > > > >
> > > > > Thanks,
> > > > > Dan
> > > > >
> > > > > On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
> > > > > grundprinzip@gmail.com> wrote:
> > > > >
> > > > > > I've seen this in the past and propose a patch to fix it:
> > > > > > https://gerrit.cloudera.org/#/c/4108/
> > > > > >
> > > > > > Let me know if you have JIRA to track this to update the commit
> > > > message.
> > > > > >
> > > > > > Martin
> > > > > >
> > > > > > On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <
> > > laszlo.gaal@cloudera.com
> > > > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > link.txt is generated by cmake based on the contents of
> > > > CMakeLists.txt
> > > > > > and
> > > > > > > the project/environment settings. Most of the link-time
> > > dependencies
> > > > > seem
> > > > > > > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on
> > > settings
> > > > > > > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > > > > > > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more
> > > into
> > > > it
> > > > > > > tomorrow.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > >     - LaszloG
> > > > > > >
> > > > > > > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <
> > > > yonghyun@cloudera.com
> > > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I observed the same linking error and digged this into
> further.
> > > The
> > > > > > below
> > > > > > > > is what I've found.
> > > > > > > >
> > > > > > > > *1. how to reproduce*
> > > > > > > >
> > > > > > > > # assuming that impala-config.sh is sourced
> > > > > > > > $ cd ${IMPALA_HOME}
> > > > > > > > *$ make -j8 impalad*
> > > > > > > > ...
> > > > > > > > [100%] Built target CodeGen
> > > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > > ...
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > `google_breakpad::ExceptionHandler::
> WriteMinidump(std::string
> > > > > const&,
> > > > > > > bool
> > > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*,
> bool),
> > > > > void*)'
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > `snappy::MaxCompressedLength(unsigned long)'
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > `my_strlen'
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > `LZ4_compress'
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > `BZ2_bzDecompress'
> > > > > > > > ...
> > > > > > > >
> > > > > > > >
> > > > > > > > *$ cd ${IMPALA_HOME}/be/src/service*
> > > > > > > > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > > > > > > > *cmake_link_script CMakeFiles/impalad.dir/link.txt
> --verbose=*
> > > > > > > > ...
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > ...
> > > > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > `LZ4_compress'*
> > > > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > `BZ2_bzDecompress'*
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > > *`snappy::RawCompress*(char const*,
> > > > > > > > unsigned long, char*, unsigned long*)'
> > > > > > > > ...
> > > > > > > > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> > > > > > > >::do_assign(char
> > > > > > > > const*, char
> > > > > > > > const*, unsigned int)'
> > > > > > > > ...
> > > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > *`snappy::*
> > > > > > > > RawUncompress(char
> > > > > > > > const*, unsigned long, char*)'
> > > > > > > >
> > > > > > > > *2. cause of the failure*
> > > > > > > >
> > > > > > > > $ cd ${IMPALA_HOME}/be/src/service
> > > > > > > > *$ cat CMakeFiles/impalad.dir/link.txt*
> > > > > > > > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall
> > > > > -Wno-sign-compare
> > > > > > > > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > > > > > > > -Wno-deprecated ...
> > > > > > > > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > > > > > > > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > > > > > > > ...
> > > > > > > > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > > > > > > > ...
> > > > > > > > *../../build/debug/util/libUtil.so*
> > > > > > > > -lpthread -ldl -lssl -lcrypto
> > > > > > > >
> > > > > > > > Apparently, libUtil.so __should__ show before libsnappy.a and
> > > > > *regex.a.
> > > > > > > >
> > > > > > > >
> > > > > > > > Next step is to find out who creates
> > CMakeFiles/impalad.dir/link.
> > > > txt
> > > > > > and
> > > > > > > > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> > > > > > > libboost_regex.a
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > > Yonghyun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <
> > > > henry@cloudera.com
> > > > > > <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Only that it's IMPALA-3979. Does anyone have any theories
> > yet?
> > > > > > > > >
> > > > > > > > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > > > > > > > dtsirogiannis@cloudera.com <javascript:;>> wrote:
> > > > > > > > >
> > > > > > > > > > Hey guys,
> > > > > > > > > >
> > > > > > > > > > When I build from the latest master using the
> > > > -build_shared_libs
> > > > > > > > option I
> > > > > > > > > > get linking errors (see [1]). Building with static works
> > > fine.
> > > > > Any
> > > > > > > > ideas?
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Dimitris
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'BZ2_bzBuffToBuffCompress'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'snappy::RawCompress(char const*, unsigned long, char*,
> > > > unsigned
> > > > > > > > long*)'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'LZ4_compress'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'BZ2_bzDecompressEnd'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'BZ2_bzDecompressInit'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'BZ2_bzBuffToBuffDecompress'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'BZ2_bzDecompress'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'snappy::GetUncompressedLength(char const*, unsigned
> long,
> > > > > > unsigned
> > > > > > > > > > long*)'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'snappy::RawUncompress(char const*, unsigned long,
> char*)'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'LZ4_uncompress'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'my_strlen'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'google_breakpad::ExceptionHandler::
> > > WriteMinidump(std::string
> > > > > > const&,
> > > > > > > > > bool
> > > > > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*,
> > > bool),
> > > > > > void*)'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > >
> > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > > > > > > > breakpad::MinidumpDescriptor
> > > > > > > > > > const&, bool (*)(void*), bool (*)(google_breakpad::
> > > > > > MinidumpDescriptor
> > > > > > > > > > const&, void*, bool), void*, bool, int)'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'ProfilerStart'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'ProfilerStop'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > > > > > > > boost::cpp_regex_traits<char> > >::do_assign(char const*,
> > > char
> > > > > > > const*,
> > > > > > > > > > unsigned int)'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > > > iterator<char
> > > > > > > > > const*,
> > > > > > > > > > std::string>,
> > > > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > > > iterator<char
> > > > > > > > > const*,
> > > > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > > > >::find()'
> > > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > > reference
> > > > to
> > > > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > > > iterator<char
> > > > > > > > > const*,
> > > > > > > > > > std::string>,
> > > > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > > > iterator<char
> > > > > > > > > const*,
> > > > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > > > >::construct_init(boost::basic_regex<char,
> > > > > > boost::regex_traits<char,
> > > > > > > > > > boost::cpp_regex_traits<char> > > const&,
> > > > > > > > > > boost::regex_constants::_match_flags)'
> > > > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use
> > > > `strerror'
> > > > > or
> > > > > > > > > > `strerror_r' instead
> > > > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > > > ldap_int_poll: warning: `sys_errlist' is deprecated; use
> > > > > `strerror'
> > > > > > > or
> > > > > > > > > > `strerror_r' instead
> > > > > > > > > > collect2: error: ld returned 1 exit status
> > > > > > > > > > be/src/service/CMakeFiles/impalad.dir/build.make:166:
> > recipe
> > > > for
> > > > > > > > target
> > > > > > > > > > 'be/build/debug/service/impalad' failed
> > > > > > > > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > > > > > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > > > > > > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all]
> > > > Error 2
> > > > > > > > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > > > > > > > make[1]: *** [be/src/service/CMakeFiles/
> impalad.dir/rule]
> > > > Error
> > > > > 2
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Henry Robinson
> > > > > > > > > Software Engineer
> > > > > > > > > Cloudera
> > > > > > > > > 415-994-6679
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>

Re: Error linking impalad with -build_shared_libs option

Posted by "Martin Grund (Das Grundprinzip.de)" <gr...@gmail.com>.
What happened is that the Util library was linked to the error-utils-test
(which it is by default anyways because of the way how ADD_BE_TEST works).
Now, what happened is that if you dynamically link the order of the
libraries plays a role. The additional target-link-library would move the
libUtil.so to the end of the library list and if the linker picks this up
some of the symbols declared earlier where already dropped.

On Wed, Aug 24, 2016 at 11:00 AM Henry Robinson <he...@cloudera.com> wrote:

> Seems great to me - but do we know why this fixes the issue? Not that that
> should prevent us from merging the patch.
>
> On 24 August 2016 at 08:54, Tim Armstrong <ta...@cloudera.com> wrote:
>
> > The fix works for me too, thanks Martin!
> >
> > I gave the patch a +1 but will upgrade to +2 if it seems everyone is
> happy
> > with it.
> >
> > Cheers,
> > Tim
> >
> > On Wed, Aug 24, 2016 at 8:15 AM, Yonghyun Hwang <yo...@cloudera.com>
> > wrote:
> >
> > > Hello Martin,
> > >
> > > Your fix works great! :) As for the build failure, what I've found is
> > that
> > > cmake 3.2.3 causes a build failure w/o your fix while cmake 2.8.12
> works
> > > good.
> > >
> > > After the commit for "IMPALA-3924: Ubuntu16 support", cmake 3.2.3 got
> > > deployed under "toolchain", which reveals this build issue.
> > >
> > > So, any one using the newer cmake would require your fix to properly
> > build
> > > impala w/ -so.
> > >
> > > Thank you,
> > > Yonghyun
> > >
> > > On Tue, Aug 23, 2016 at 11:40 PM, Daniel Hecht <dh...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Martin,
> > > >
> > > > Nice to hear from you.
> https://issues.cloudera.org/browse/IMPALA-3979
> > > >
> > > > Thanks,
> > > > Dan
> > > >
> > > > On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
> > > > grundprinzip@gmail.com> wrote:
> > > >
> > > > > I've seen this in the past and propose a patch to fix it:
> > > > > https://gerrit.cloudera.org/#/c/4108/
> > > > >
> > > > > Let me know if you have JIRA to track this to update the commit
> > > message.
> > > > >
> > > > > Martin
> > > > >
> > > > > On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <
> > laszlo.gaal@cloudera.com
> > > > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > link.txt is generated by cmake based on the contents of
> > > CMakeLists.txt
> > > > > and
> > > > > > the project/environment settings. Most of the link-time
> > dependencies
> > > > seem
> > > > > > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on
> > settings
> > > > > > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > > > > > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more
> > into
> > > it
> > > > > > tomorrow.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > >     - LaszloG
> > > > > >
> > > > > > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <
> > > yonghyun@cloudera.com
> > > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > I observed the same linking error and digged this into further.
> > The
> > > > > below
> > > > > > > is what I've found.
> > > > > > >
> > > > > > > *1. how to reproduce*
> > > > > > >
> > > > > > > # assuming that impala-config.sh is sourced
> > > > > > > $ cd ${IMPALA_HOME}
> > > > > > > *$ make -j8 impalad*
> > > > > > > ...
> > > > > > > [100%] Built target CodeGen
> > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > ...
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > `google_breakpad::ExceptionHandler::WriteMinidump(std::string
> > > > const&,
> > > > > > bool
> > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool),
> > > > void*)'
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > `snappy::MaxCompressedLength(unsigned long)'
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > `my_strlen'
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > `LZ4_compress'
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > `BZ2_bzDecompress'
> > > > > > > ...
> > > > > > >
> > > > > > >
> > > > > > > *$ cd ${IMPALA_HOME}/be/src/service*
> > > > > > > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > > > > > > *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> > > > > > > ...
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > ...
> > > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > > `LZ4_compress'*
> > > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > `BZ2_bzDecompress'*
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > *`snappy::RawCompress*(char const*,
> > > > > > > unsigned long, char*, unsigned long*)'
> > > > > > > ...
> > > > > > > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> > > > > > >::do_assign(char
> > > > > > > const*, char
> > > > > > > const*, unsigned int)'
> > > > > > > ...
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > *`snappy::*
> > > > > > > RawUncompress(char
> > > > > > > const*, unsigned long, char*)'
> > > > > > >
> > > > > > > *2. cause of the failure*
> > > > > > >
> > > > > > > $ cd ${IMPALA_HOME}/be/src/service
> > > > > > > *$ cat CMakeFiles/impalad.dir/link.txt*
> > > > > > > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall
> > > > -Wno-sign-compare
> > > > > > > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > > > > > > -Wno-deprecated ...
> > > > > > > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > > > > > > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > > > > > > ...
> > > > > > > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > > > > > > ...
> > > > > > > *../../build/debug/util/libUtil.so*
> > > > > > > -lpthread -ldl -lssl -lcrypto
> > > > > > >
> > > > > > > Apparently, libUtil.so __should__ show before libsnappy.a and
> > > > *regex.a.
> > > > > > >
> > > > > > >
> > > > > > > Next step is to find out who creates
> CMakeFiles/impalad.dir/link.
> > > txt
> > > > > and
> > > > > > > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> > > > > > libboost_regex.a
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Yonghyun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <
> > > henry@cloudera.com
> > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Only that it's IMPALA-3979. Does anyone have any theories
> yet?
> > > > > > > >
> > > > > > > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > > > > > > dtsirogiannis@cloudera.com <javascript:;>> wrote:
> > > > > > > >
> > > > > > > > > Hey guys,
> > > > > > > > >
> > > > > > > > > When I build from the latest master using the
> > > -build_shared_libs
> > > > > > > option I
> > > > > > > > > get linking errors (see [1]). Building with static works
> > fine.
> > > > Any
> > > > > > > ideas?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Dimitris
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'BZ2_bzBuffToBuffCompress'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'snappy::RawCompress(char const*, unsigned long, char*,
> > > unsigned
> > > > > > > long*)'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'LZ4_compress'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'BZ2_bzDecompressEnd'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'BZ2_bzDecompressInit'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'BZ2_bzBuffToBuffDecompress'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'BZ2_bzDecompress'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'snappy::GetUncompressedLength(char const*, unsigned long,
> > > > > unsigned
> > > > > > > > > long*)'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'LZ4_uncompress'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'my_strlen'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'google_breakpad::ExceptionHandler::
> > WriteMinidump(std::string
> > > > > const&,
> > > > > > > > bool
> > > > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*,
> > bool),
> > > > > void*)'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > >
> 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > > > > > > breakpad::MinidumpDescriptor
> > > > > > > > > const&, bool (*)(void*), bool (*)(google_breakpad::
> > > > > MinidumpDescriptor
> > > > > > > > > const&, void*, bool), void*, bool, int)'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'ProfilerStart'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'ProfilerStop'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > > > > > > boost::cpp_regex_traits<char> > >::do_assign(char const*,
> > char
> > > > > > const*,
> > > > > > > > > unsigned int)'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > > iterator<char
> > > > > > > > const*,
> > > > > > > > > std::string>,
> > > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > > iterator<char
> > > > > > > > const*,
> > > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > > >::find()'
> > > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> > reference
> > > to
> > > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > > iterator<char
> > > > > > > > const*,
> > > > > > > > > std::string>,
> > > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > > iterator<char
> > > > > > > > const*,
> > > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > > >::construct_init(boost::basic_regex<char,
> > > > > boost::regex_traits<char,
> > > > > > > > > boost::cpp_regex_traits<char> > > const&,
> > > > > > > > > boost::regex_constants::_match_flags)'
> > > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use
> > > `strerror'
> > > > or
> > > > > > > > > `strerror_r' instead
> > > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > > ldap_int_poll: warning: `sys_errlist' is deprecated; use
> > > > `strerror'
> > > > > > or
> > > > > > > > > `strerror_r' instead
> > > > > > > > > collect2: error: ld returned 1 exit status
> > > > > > > > > be/src/service/CMakeFiles/impalad.dir/build.make:166:
> recipe
> > > for
> > > > > > > target
> > > > > > > > > 'be/build/debug/service/impalad' failed
> > > > > > > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > > > > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > > > > > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all]
> > > Error 2
> > > > > > > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > > > > > > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule]
> > > Error
> > > > 2
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Henry Robinson
> > > > > > > > Software Engineer
> > > > > > > > Cloudera
> > > > > > > > 415-994-6679
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>

Re: Error linking impalad with -build_shared_libs option

Posted by Henry Robinson <he...@cloudera.com>.
Seems great to me - but do we know why this fixes the issue? Not that that
should prevent us from merging the patch.

On 24 August 2016 at 08:54, Tim Armstrong <ta...@cloudera.com> wrote:

> The fix works for me too, thanks Martin!
>
> I gave the patch a +1 but will upgrade to +2 if it seems everyone is happy
> with it.
>
> Cheers,
> Tim
>
> On Wed, Aug 24, 2016 at 8:15 AM, Yonghyun Hwang <yo...@cloudera.com>
> wrote:
>
> > Hello Martin,
> >
> > Your fix works great! :) As for the build failure, what I've found is
> that
> > cmake 3.2.3 causes a build failure w/o your fix while cmake 2.8.12 works
> > good.
> >
> > After the commit for "IMPALA-3924: Ubuntu16 support", cmake 3.2.3 got
> > deployed under "toolchain", which reveals this build issue.
> >
> > So, any one using the newer cmake would require your fix to properly
> build
> > impala w/ -so.
> >
> > Thank you,
> > Yonghyun
> >
> > On Tue, Aug 23, 2016 at 11:40 PM, Daniel Hecht <dh...@cloudera.com>
> > wrote:
> >
> > > Hi Martin,
> > >
> > > Nice to hear from you. https://issues.cloudera.org/browse/IMPALA-3979
> > >
> > > Thanks,
> > > Dan
> > >
> > > On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
> > > grundprinzip@gmail.com> wrote:
> > >
> > > > I've seen this in the past and propose a patch to fix it:
> > > > https://gerrit.cloudera.org/#/c/4108/
> > > >
> > > > Let me know if you have JIRA to track this to update the commit
> > message.
> > > >
> > > > Martin
> > > >
> > > > On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <
> laszlo.gaal@cloudera.com
> > > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > link.txt is generated by cmake based on the contents of
> > CMakeLists.txt
> > > > and
> > > > > the project/environment settings. Most of the link-time
> dependencies
> > > seem
> > > > > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on
> settings
> > > > > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > > > > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more
> into
> > it
> > > > > tomorrow.
> > > > >
> > > > > Thanks,
> > > > >
> > > > >     - LaszloG
> > > > >
> > > > > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <
> > yonghyun@cloudera.com
> > > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > I observed the same linking error and digged this into further.
> The
> > > > below
> > > > > > is what I've found.
> > > > > >
> > > > > > *1. how to reproduce*
> > > > > >
> > > > > > # assuming that impala-config.sh is sourced
> > > > > > $ cd ${IMPALA_HOME}
> > > > > > *$ make -j8 impalad*
> > > > > > ...
> > > > > > [100%] Built target CodeGen
> > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > ...
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > `google_breakpad::ExceptionHandler::WriteMinidump(std::string
> > > const&,
> > > > > bool
> > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool),
> > > void*)'
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > `snappy::MaxCompressedLength(unsigned long)'
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > `my_strlen'
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > `LZ4_compress'
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > `BZ2_bzDecompress'
> > > > > > ...
> > > > > >
> > > > > >
> > > > > > *$ cd ${IMPALA_HOME}/be/src/service*
> > > > > > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > > > > > *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> > > > > > ...
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > ...
> > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > `LZ4_compress'*
> > > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > > `BZ2_bzDecompress'*
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > *`snappy::RawCompress*(char const*,
> > > > > > unsigned long, char*, unsigned long*)'
> > > > > > ...
> > > > > > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> > > > > >::do_assign(char
> > > > > > const*, char
> > > > > > const*, unsigned int)'
> > > > > > ...
> > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > *`snappy::*
> > > > > > RawUncompress(char
> > > > > > const*, unsigned long, char*)'
> > > > > >
> > > > > > *2. cause of the failure*
> > > > > >
> > > > > > $ cd ${IMPALA_HOME}/be/src/service
> > > > > > *$ cat CMakeFiles/impalad.dir/link.txt*
> > > > > > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall
> > > -Wno-sign-compare
> > > > > > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > > > > > -Wno-deprecated ...
> > > > > > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > > > > > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > > > > > ...
> > > > > > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > > > > > ...
> > > > > > *../../build/debug/util/libUtil.so*
> > > > > > -lpthread -ldl -lssl -lcrypto
> > > > > >
> > > > > > Apparently, libUtil.so __should__ show before libsnappy.a and
> > > *regex.a.
> > > > > >
> > > > > >
> > > > > > Next step is to find out who creates CMakeFiles/impalad.dir/link.
> > txt
> > > > and
> > > > > > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> > > > > libboost_regex.a
> > > > > >
> > > > > > Thank you,
> > > > > > Yonghyun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <
> > henry@cloudera.com
> > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > Only that it's IMPALA-3979. Does anyone have any theories yet?
> > > > > > >
> > > > > > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > > > > > dtsirogiannis@cloudera.com <javascript:;>> wrote:
> > > > > > >
> > > > > > > > Hey guys,
> > > > > > > >
> > > > > > > > When I build from the latest master using the
> > -build_shared_libs
> > > > > > option I
> > > > > > > > get linking errors (see [1]). Building with static works
> fine.
> > > Any
> > > > > > ideas?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Dimitris
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]
> > > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'BZ2_bzBuffToBuffCompress'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'snappy::RawCompress(char const*, unsigned long, char*,
> > unsigned
> > > > > > long*)'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'LZ4_compress'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'BZ2_bzDecompressEnd'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'BZ2_bzDecompressInit'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'BZ2_bzBuffToBuffDecompress'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'BZ2_bzDecompress'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'snappy::GetUncompressedLength(char const*, unsigned long,
> > > > unsigned
> > > > > > > > long*)'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'LZ4_uncompress'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'my_strlen'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'google_breakpad::ExceptionHandler::
> WriteMinidump(std::string
> > > > const&,
> > > > > > > bool
> > > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*,
> bool),
> > > > void*)'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > > > > > breakpad::MinidumpDescriptor
> > > > > > > > const&, bool (*)(void*), bool (*)(google_breakpad::
> > > > MinidumpDescriptor
> > > > > > > > const&, void*, bool), void*, bool, int)'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'ProfilerStart'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'ProfilerStop'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > > > > > boost::cpp_regex_traits<char> > >::do_assign(char const*,
> char
> > > > > const*,
> > > > > > > > unsigned int)'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > iterator<char
> > > > > > > const*,
> > > > > > > > std::string>,
> > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > iterator<char
> > > > > > > const*,
> > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > >::find()'
> > > > > > > > ../../build/debug/util/libUtil.so: error: undefined
> reference
> > to
> > > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > > iterator<char
> > > > > > > const*,
> > > > > > > > std::string>,
> > > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > > iterator<char
> > > > > > > const*,
> > > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > > boost::cpp_regex_traits<char> >
> > > > > > > > >::construct_init(boost::basic_regex<char,
> > > > boost::regex_traits<char,
> > > > > > > > boost::cpp_regex_traits<char> > > const&,
> > > > > > > > boost::regex_constants::_match_flags)'
> > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use
> > `strerror'
> > > or
> > > > > > > > `strerror_r' instead
> > > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > > os-ip.o):os-ip.c:function
> > > > > > > > ldap_int_poll: warning: `sys_errlist' is deprecated; use
> > > `strerror'
> > > > > or
> > > > > > > > `strerror_r' instead
> > > > > > > > collect2: error: ld returned 1 exit status
> > > > > > > > be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe
> > for
> > > > > > target
> > > > > > > > 'be/build/debug/service/impalad' failed
> > > > > > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > > > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > > > > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all]
> > Error 2
> > > > > > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > > > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > > > > > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule]
> > Error
> > > 2
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Henry Robinson
> > > > > > > Software Engineer
> > > > > > > Cloudera
> > > > > > > 415-994-6679
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Re: Error linking impalad with -build_shared_libs option

Posted by Tim Armstrong <ta...@cloudera.com>.
The fix works for me too, thanks Martin!

I gave the patch a +1 but will upgrade to +2 if it seems everyone is happy
with it.

Cheers,
Tim

On Wed, Aug 24, 2016 at 8:15 AM, Yonghyun Hwang <yo...@cloudera.com>
wrote:

> Hello Martin,
>
> Your fix works great! :) As for the build failure, what I've found is that
> cmake 3.2.3 causes a build failure w/o your fix while cmake 2.8.12 works
> good.
>
> After the commit for "IMPALA-3924: Ubuntu16 support", cmake 3.2.3 got
> deployed under "toolchain", which reveals this build issue.
>
> So, any one using the newer cmake would require your fix to properly build
> impala w/ -so.
>
> Thank you,
> Yonghyun
>
> On Tue, Aug 23, 2016 at 11:40 PM, Daniel Hecht <dh...@cloudera.com>
> wrote:
>
> > Hi Martin,
> >
> > Nice to hear from you. https://issues.cloudera.org/browse/IMPALA-3979
> >
> > Thanks,
> > Dan
> >
> > On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
> > grundprinzip@gmail.com> wrote:
> >
> > > I've seen this in the past and propose a patch to fix it:
> > > https://gerrit.cloudera.org/#/c/4108/
> > >
> > > Let me know if you have JIRA to track this to update the commit
> message.
> > >
> > > Martin
> > >
> > > On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <laszlo.gaal@cloudera.com
> > > <javascript:;>>
> > > wrote:
> > >
> > > > link.txt is generated by cmake based on the contents of
> CMakeLists.txt
> > > and
> > > > the project/environment settings. Most of the link-time dependencies
> > seem
> > > > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on settings
> > > > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > > > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more into
> it
> > > > tomorrow.
> > > >
> > > > Thanks,
> > > >
> > > >     - LaszloG
> > > >
> > > > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <
> yonghyun@cloudera.com
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > I observed the same linking error and digged this into further. The
> > > below
> > > > > is what I've found.
> > > > >
> > > > > *1. how to reproduce*
> > > > >
> > > > > # assuming that impala-config.sh is sourced
> > > > > $ cd ${IMPALA_HOME}
> > > > > *$ make -j8 impalad*
> > > > > ...
> > > > > [100%] Built target CodeGen
> > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > ...
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > `google_breakpad::ExceptionHandler::WriteMinidump(std::string
> > const&,
> > > > bool
> > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool),
> > void*)'
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > `snappy::MaxCompressedLength(unsigned long)'
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > `my_strlen'
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > `LZ4_compress'
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > `BZ2_bzDecompress'
> > > > > ...
> > > > >
> > > > >
> > > > > *$ cd ${IMPALA_HOME}/be/src/service*
> > > > > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > > > > *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> > > > > ...
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > ...
> > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > `LZ4_compress'*
> > > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > > `BZ2_bzDecompress'*
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > *`snappy::RawCompress*(char const*,
> > > > > unsigned long, char*, unsigned long*)'
> > > > > ...
> > > > > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> > > > >::do_assign(char
> > > > > const*, char
> > > > > const*, unsigned int)'
> > > > > ...
> > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > *`snappy::*
> > > > > RawUncompress(char
> > > > > const*, unsigned long, char*)'
> > > > >
> > > > > *2. cause of the failure*
> > > > >
> > > > > $ cd ${IMPALA_HOME}/be/src/service
> > > > > *$ cat CMakeFiles/impalad.dir/link.txt*
> > > > > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall
> > -Wno-sign-compare
> > > > > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > > > > -Wno-deprecated ...
> > > > > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > > > > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > > > > ...
> > > > > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > > > > ...
> > > > > *../../build/debug/util/libUtil.so*
> > > > > -lpthread -ldl -lssl -lcrypto
> > > > >
> > > > > Apparently, libUtil.so __should__ show before libsnappy.a and
> > *regex.a.
> > > > >
> > > > >
> > > > > Next step is to find out who creates CMakeFiles/impalad.dir/link.
> txt
> > > and
> > > > > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> > > > libboost_regex.a
> > > > >
> > > > > Thank you,
> > > > > Yonghyun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <
> henry@cloudera.com
> > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > Only that it's IMPALA-3979. Does anyone have any theories yet?
> > > > > >
> > > > > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > > > > dtsirogiannis@cloudera.com <javascript:;>> wrote:
> > > > > >
> > > > > > > Hey guys,
> > > > > > >
> > > > > > > When I build from the latest master using the
> -build_shared_libs
> > > > > option I
> > > > > > > get linking errors (see [1]). Building with static works fine.
> > Any
> > > > > ideas?
> > > > > > >
> > > > > > > Thanks
> > > > > > > Dimitris
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'BZ2_bzBuffToBuffCompress'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'snappy::RawCompress(char const*, unsigned long, char*,
> unsigned
> > > > > long*)'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'LZ4_compress'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'BZ2_bzDecompressEnd'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'BZ2_bzDecompressInit'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'BZ2_bzBuffToBuffDecompress'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'BZ2_bzDecompress'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'snappy::GetUncompressedLength(char const*, unsigned long,
> > > unsigned
> > > > > > > long*)'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'LZ4_uncompress'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'my_strlen'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'google_breakpad::ExceptionHandler::WriteMinidump(std::string
> > > const&,
> > > > > > bool
> > > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool),
> > > void*)'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > > > > breakpad::MinidumpDescriptor
> > > > > > > const&, bool (*)(void*), bool (*)(google_breakpad::
> > > MinidumpDescriptor
> > > > > > > const&, void*, bool), void*, bool, int)'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'ProfilerStart'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'ProfilerStop'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > > > > boost::cpp_regex_traits<char> > >::do_assign(char const*, char
> > > > const*,
> > > > > > > unsigned int)'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > iterator<char
> > > > > > const*,
> > > > > > > std::string>,
> > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > iterator<char
> > > > > > const*,
> > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > boost::cpp_regex_traits<char> >
> > > > > > > >::find()'
> > > > > > > ../../build/debug/util/libUtil.so: error: undefined reference
> to
> > > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> > iterator<char
> > > > > > const*,
> > > > > > > std::string>,
> > > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> > iterator<char
> > > > > > const*,
> > > > > > > std::string> > >, boost::regex_traits<char,
> > > > > > boost::cpp_regex_traits<char> >
> > > > > > > >::construct_init(boost::basic_regex<char,
> > > boost::regex_traits<char,
> > > > > > > boost::cpp_regex_traits<char> > > const&,
> > > > > > > boost::regex_constants::_match_flags)'
> > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > os-ip.o):os-ip.c:function
> > > > > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use
> `strerror'
> > or
> > > > > > > `strerror_r' instead
> > > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > > os-ip.o):os-ip.c:function
> > > > > > > ldap_int_poll: warning: `sys_errlist' is deprecated; use
> > `strerror'
> > > > or
> > > > > > > `strerror_r' instead
> > > > > > > collect2: error: ld returned 1 exit status
> > > > > > > be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe
> for
> > > > > target
> > > > > > > 'be/build/debug/service/impalad' failed
> > > > > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > > > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all]
> Error 2
> > > > > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > > > > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule]
> Error
> > 2
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Henry Robinson
> > > > > > Software Engineer
> > > > > > Cloudera
> > > > > > 415-994-6679
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Error linking impalad with -build_shared_libs option

Posted by Yonghyun Hwang <yo...@cloudera.com>.
Hello Martin,

Your fix works great! :) As for the build failure, what I've found is that
cmake 3.2.3 causes a build failure w/o your fix while cmake 2.8.12 works
good.

After the commit for "IMPALA-3924: Ubuntu16 support", cmake 3.2.3 got
deployed under "toolchain", which reveals this build issue.

So, any one using the newer cmake would require your fix to properly build
impala w/ -so.

Thank you,
Yonghyun

On Tue, Aug 23, 2016 at 11:40 PM, Daniel Hecht <dh...@cloudera.com> wrote:

> Hi Martin,
>
> Nice to hear from you. https://issues.cloudera.org/browse/IMPALA-3979
>
> Thanks,
> Dan
>
> On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
> grundprinzip@gmail.com> wrote:
>
> > I've seen this in the past and propose a patch to fix it:
> > https://gerrit.cloudera.org/#/c/4108/
> >
> > Let me know if you have JIRA to track this to update the commit message.
> >
> > Martin
> >
> > On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <laszlo.gaal@cloudera.com
> > <javascript:;>>
> > wrote:
> >
> > > link.txt is generated by cmake based on the contents of CMakeLists.txt
> > and
> > > the project/environment settings. Most of the link-time dependencies
> seem
> > > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on settings
> > > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more into it
> > > tomorrow.
> > >
> > > Thanks,
> > >
> > >     - LaszloG
> > >
> > > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <yonghyun@cloudera.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > I observed the same linking error and digged this into further. The
> > below
> > > > is what I've found.
> > > >
> > > > *1. how to reproduce*
> > > >
> > > > # assuming that impala-config.sh is sourced
> > > > $ cd ${IMPALA_HOME}
> > > > *$ make -j8 impalad*
> > > > ...
> > > > [100%] Built target CodeGen
> > > > Linking CXX executable ../../build/debug/service/impalad
> > > > ...
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > `google_breakpad::ExceptionHandler::WriteMinidump(std::string
> const&,
> > > bool
> > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool),
> void*)'
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > `snappy::MaxCompressedLength(unsigned long)'
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> `my_strlen'
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> > `LZ4_compress'
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > `BZ2_bzDecompress'
> > > > ...
> > > >
> > > >
> > > > *$ cd ${IMPALA_HOME}/be/src/service*
> > > > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > > > *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> > > > ...
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > ...
> > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > `LZ4_compress'*
> > > > *../../build/debug/util/libUtil.so: undefined reference to
> > > > `BZ2_bzDecompress'*
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > *`snappy::RawCompress*(char const*,
> > > > unsigned long, char*, unsigned long*)'
> > > > ...
> > > > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> > > >::do_assign(char
> > > > const*, char
> > > > const*, unsigned int)'
> > > > ...
> > > > ../../build/debug/util/libUtil.so: undefined reference to
> *`snappy::*
> > > > RawUncompress(char
> > > > const*, unsigned long, char*)'
> > > >
> > > > *2. cause of the failure*
> > > >
> > > > $ cd ${IMPALA_HOME}/be/src/service
> > > > *$ cat CMakeFiles/impalad.dir/link.txt*
> > > > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall
> -Wno-sign-compare
> > > > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > > > -Wno-deprecated ...
> > > > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > > > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > > > ...
> > > > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > > > ...
> > > > *../../build/debug/util/libUtil.so*
> > > > -lpthread -ldl -lssl -lcrypto
> > > >
> > > > Apparently, libUtil.so __should__ show before libsnappy.a and
> *regex.a.
> > > >
> > > >
> > > > Next step is to find out who creates CMakeFiles/impalad.dir/link.txt
> > and
> > > > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> > > libboost_regex.a
> > > >
> > > > Thank you,
> > > > Yonghyun
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <henry@cloudera.com
> > <javascript:;>>
> > > > wrote:
> > > >
> > > > > Only that it's IMPALA-3979. Does anyone have any theories yet?
> > > > >
> > > > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > > > dtsirogiannis@cloudera.com <javascript:;>> wrote:
> > > > >
> > > > > > Hey guys,
> > > > > >
> > > > > > When I build from the latest master using the -build_shared_libs
> > > > option I
> > > > > > get linking errors (see [1]). Building with static works fine.
> Any
> > > > ideas?
> > > > > >
> > > > > > Thanks
> > > > > > Dimitris
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'BZ2_bzBuffToBuffCompress'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'snappy::RawCompress(char const*, unsigned long, char*, unsigned
> > > > long*)'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'LZ4_compress'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'BZ2_bzDecompressEnd'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'BZ2_bzDecompressInit'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'BZ2_bzBuffToBuffDecompress'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'BZ2_bzDecompress'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'snappy::GetUncompressedLength(char const*, unsigned long,
> > unsigned
> > > > > > long*)'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'LZ4_uncompress'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'my_strlen'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'google_breakpad::ExceptionHandler::WriteMinidump(std::string
> > const&,
> > > > > bool
> > > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool),
> > void*)'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > > > breakpad::MinidumpDescriptor
> > > > > > const&, bool (*)(void*), bool (*)(google_breakpad::
> > MinidumpDescriptor
> > > > > > const&, void*, bool), void*, bool, int)'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'ProfilerStart'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'ProfilerStop'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > > > boost::cpp_regex_traits<char> > >::do_assign(char const*, char
> > > const*,
> > > > > > unsigned int)'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> iterator<char
> > > > > const*,
> > > > > > std::string>,
> > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> iterator<char
> > > > > const*,
> > > > > > std::string> > >, boost::regex_traits<char,
> > > > > boost::cpp_regex_traits<char> >
> > > > > > >::find()'
> > > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_
> iterator<char
> > > > > const*,
> > > > > > std::string>,
> > > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_
> iterator<char
> > > > > const*,
> > > > > > std::string> > >, boost::regex_traits<char,
> > > > > boost::cpp_regex_traits<char> >
> > > > > > >::construct_init(boost::basic_regex<char,
> > boost::regex_traits<char,
> > > > > > boost::cpp_regex_traits<char> > > const&,
> > > > > > boost::regex_constants::_match_flags)'
> > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > os-ip.o):os-ip.c:function
> > > > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror'
> or
> > > > > > `strerror_r' instead
> > > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > > os-ip.o):os-ip.c:function
> > > > > > ldap_int_poll: warning: `sys_errlist' is deprecated; use
> `strerror'
> > > or
> > > > > > `strerror_r' instead
> > > > > > collect2: error: ld returned 1 exit status
> > > > > > be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe for
> > > > target
> > > > > > 'be/build/debug/service/impalad' failed
> > > > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
> > > > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > > > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error
> 2
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Henry Robinson
> > > > > Software Engineer
> > > > > Cloudera
> > > > > 415-994-6679
> > > > >
> > > >
> > >
> >
>

Re: Error linking impalad with -build_shared_libs option

Posted by Daniel Hecht <dh...@cloudera.com>.
Hi Martin,

Nice to hear from you. https://issues.cloudera.org/browse/IMPALA-3979

Thanks,
Dan

On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
grundprinzip@gmail.com> wrote:

> I've seen this in the past and propose a patch to fix it:
> https://gerrit.cloudera.org/#/c/4108/
>
> Let me know if you have JIRA to track this to update the commit message.
>
> Martin
>
> On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <laszlo.gaal@cloudera.com
> <javascript:;>>
> wrote:
>
> > link.txt is generated by cmake based on the contents of CMakeLists.txt
> and
> > the project/environment settings. Most of the link-time dependencies seem
> > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on settings
> > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more into it
> > tomorrow.
> >
> > Thanks,
> >
> >     - LaszloG
> >
> > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <yonghyun@cloudera.com
> <javascript:;>>
> > wrote:
> >
> > > I observed the same linking error and digged this into further. The
> below
> > > is what I've found.
> > >
> > > *1. how to reproduce*
> > >
> > > # assuming that impala-config.sh is sourced
> > > $ cd ${IMPALA_HOME}
> > > *$ make -j8 impalad*
> > > ...
> > > [100%] Built target CodeGen
> > > Linking CXX executable ../../build/debug/service/impalad
> > > ...
> > > ../../build/debug/util/libUtil.so: undefined reference to
> > > `google_breakpad::ExceptionHandler::WriteMinidump(std::string const&,
> > bool
> > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> > > ../../build/debug/util/libUtil.so: undefined reference to
> > > `snappy::MaxCompressedLength(unsigned long)'
> > > ../../build/debug/util/libUtil.so: undefined reference to `my_strlen'
> > > ../../build/debug/util/libUtil.so: undefined reference to
> `LZ4_compress'
> > > ../../build/debug/util/libUtil.so: undefined reference to
> > > `BZ2_bzDecompress'
> > > ...
> > >
> > >
> > > *$ cd ${IMPALA_HOME}/be/src/service*
> > > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > > *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> > > ...
> > > ../../build/debug/util/libUtil.so: undefined reference to
> > > ...
> > > *../../build/debug/util/libUtil.so: undefined reference to
> > `LZ4_compress'*
> > > *../../build/debug/util/libUtil.so: undefined reference to
> > > `BZ2_bzDecompress'*
> > > ../../build/debug/util/libUtil.so: undefined reference to
> > > *`snappy::RawCompress*(char const*,
> > > unsigned long, char*, unsigned long*)'
> > > ...
> > > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> > >::do_assign(char
> > > const*, char
> > > const*, unsigned int)'
> > > ...
> > > ../../build/debug/util/libUtil.so: undefined reference to *`snappy::*
> > > RawUncompress(char
> > > const*, unsigned long, char*)'
> > >
> > > *2. cause of the failure*
> > >
> > > $ cd ${IMPALA_HOME}/be/src/service
> > > *$ cat CMakeFiles/impalad.dir/link.txt*
> > > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall -Wno-sign-compare
> > > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > > -Wno-deprecated ...
> > > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > > ...
> > > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > > ...
> > > *../../build/debug/util/libUtil.so*
> > > -lpthread -ldl -lssl -lcrypto
> > >
> > > Apparently, libUtil.so __should__ show before libsnappy.a and *regex.a.
> > >
> > >
> > > Next step is to find out who creates CMakeFiles/impalad.dir/link.txt
> and
> > > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> > libboost_regex.a
> > >
> > > Thank you,
> > > Yonghyun
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <henry@cloudera.com
> <javascript:;>>
> > > wrote:
> > >
> > > > Only that it's IMPALA-3979. Does anyone have any theories yet?
> > > >
> > > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > > dtsirogiannis@cloudera.com <javascript:;>> wrote:
> > > >
> > > > > Hey guys,
> > > > >
> > > > > When I build from the latest master using the -build_shared_libs
> > > option I
> > > > > get linking errors (see [1]). Building with static works fine. Any
> > > ideas?
> > > > >
> > > > > Thanks
> > > > > Dimitris
> > > > >
> > > > >
> > > > > [1]
> > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'BZ2_bzBuffToBuffCompress'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'snappy::RawCompress(char const*, unsigned long, char*, unsigned
> > > long*)'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'LZ4_compress'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'BZ2_bzDecompressEnd'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'BZ2_bzDecompressInit'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'BZ2_bzBuffToBuffDecompress'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'BZ2_bzDecompress'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'snappy::GetUncompressedLength(char const*, unsigned long,
> unsigned
> > > > > long*)'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'LZ4_uncompress'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'my_strlen'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'google_breakpad::ExceptionHandler::WriteMinidump(std::string
> const&,
> > > > bool
> > > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool),
> void*)'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > > breakpad::MinidumpDescriptor
> > > > > const&, bool (*)(void*), bool (*)(google_breakpad::
> MinidumpDescriptor
> > > > > const&, void*, bool), void*, bool, int)'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'ProfilerStart'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'ProfilerStop'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > > boost::cpp_regex_traits<char> > >::do_assign(char const*, char
> > const*,
> > > > > unsigned int)'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> > > > const*,
> > > > > std::string>,
> > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> > > > const*,
> > > > > std::string> > >, boost::regex_traits<char,
> > > > boost::cpp_regex_traits<char> >
> > > > > >::find()'
> > > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> > > > const*,
> > > > > std::string>,
> > > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> > > > const*,
> > > > > std::string> > >, boost::regex_traits<char,
> > > > boost::cpp_regex_traits<char> >
> > > > > >::construct_init(boost::basic_regex<char,
> boost::regex_traits<char,
> > > > > boost::cpp_regex_traits<char> > > const&,
> > > > > boost::regex_constants::_match_flags)'
> > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > os-ip.o):os-ip.c:function
> > > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or
> > > > > `strerror_r' instead
> > > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > > os-ip.o):os-ip.c:function
> > > > > ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror'
> > or
> > > > > `strerror_r' instead
> > > > > collect2: error: ld returned 1 exit status
> > > > > be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe for
> > > target
> > > > > 'be/build/debug/service/impalad' failed
> > > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
> > > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Henry Robinson
> > > > Software Engineer
> > > > Cloudera
> > > > 415-994-6679
> > > >
> > >
> >
>

Re: Error linking impalad with -build_shared_libs option

Posted by "Martin Grund (Das Grundprinzip.de)" <gr...@gmail.com>.
I've seen this in the past and propose a patch to fix it:
https://gerrit.cloudera.org/#/c/4108/

Let me know if you have JIRA to track this to update the commit message.

Martin

On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <la...@cloudera.com>
wrote:

> link.txt is generated by cmake based on the contents of CMakeLists.txt and
> the project/environment settings. Most of the link-time dependencies seem
> to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on settings
> calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more into it
> tomorrow.
>
> Thanks,
>
>     - LaszloG
>
> On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <yo...@cloudera.com>
> wrote:
>
> > I observed the same linking error and digged this into further. The below
> > is what I've found.
> >
> > *1. how to reproduce*
> >
> > # assuming that impala-config.sh is sourced
> > $ cd ${IMPALA_HOME}
> > *$ make -j8 impalad*
> > ...
> > [100%] Built target CodeGen
> > Linking CXX executable ../../build/debug/service/impalad
> > ...
> > ../../build/debug/util/libUtil.so: undefined reference to
> > `google_breakpad::ExceptionHandler::WriteMinidump(std::string const&,
> bool
> > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> > ../../build/debug/util/libUtil.so: undefined reference to
> > `snappy::MaxCompressedLength(unsigned long)'
> > ../../build/debug/util/libUtil.so: undefined reference to `my_strlen'
> > ../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'
> > ../../build/debug/util/libUtil.so: undefined reference to
> > `BZ2_bzDecompress'
> > ...
> >
> >
> > *$ cd ${IMPALA_HOME}/be/src/service*
> > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> > ...
> > ../../build/debug/util/libUtil.so: undefined reference to
> > ...
> > *../../build/debug/util/libUtil.so: undefined reference to
> `LZ4_compress'*
> > *../../build/debug/util/libUtil.so: undefined reference to
> > `BZ2_bzDecompress'*
> > ../../build/debug/util/libUtil.so: undefined reference to
> > *`snappy::RawCompress*(char const*,
> > unsigned long, char*, unsigned long*)'
> > ...
> > boost::regex_traits<char, boost::cpp_regex_traits<char> >
> >::do_assign(char
> > const*, char
> > const*, unsigned int)'
> > ...
> > ../../build/debug/util/libUtil.so: undefined reference to *`snappy::*
> > RawUncompress(char
> > const*, unsigned long, char*)'
> >
> > *2. cause of the failure*
> >
> > $ cd ${IMPALA_HOME}/be/src/service
> > *$ cat CMakeFiles/impalad.dir/link.txt*
> > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall -Wno-sign-compare
> > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > -Wno-deprecated ...
> > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > ...
> > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > ...
> > *../../build/debug/util/libUtil.so*
> > -lpthread -ldl -lssl -lcrypto
> >
> > Apparently, libUtil.so __should__ show before libsnappy.a and *regex.a.
> >
> >
> > Next step is to find out who creates CMakeFiles/impalad.dir/link.txt and
> > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> libboost_regex.a
> >
> > Thank you,
> > Yonghyun
> >
> >
> >
> >
> >
> > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <he...@cloudera.com>
> > wrote:
> >
> > > Only that it's IMPALA-3979. Does anyone have any theories yet?
> > >
> > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > dtsirogiannis@cloudera.com> wrote:
> > >
> > > > Hey guys,
> > > >
> > > > When I build from the latest master using the -build_shared_libs
> > option I
> > > > get linking errors (see [1]). Building with static works fine. Any
> > ideas?
> > > >
> > > > Thanks
> > > > Dimitris
> > > >
> > > >
> > > > [1]
> > > > Linking CXX executable ../../build/debug/service/impalad
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzBuffToBuffCompress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'snappy::RawCompress(char const*, unsigned long, char*, unsigned
> > long*)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'LZ4_compress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzDecompressEnd'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzDecompressInit'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzBuffToBuffDecompress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzDecompress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'snappy::GetUncompressedLength(char const*, unsigned long, unsigned
> > > > long*)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'LZ4_uncompress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'my_strlen'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'google_breakpad::ExceptionHandler::WriteMinidump(std::string const&,
> > > bool
> > > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > > breakpad::MinidumpDescriptor
> > > > const&, bool (*)(void*), bool (*)(google_breakpad::MinidumpDescriptor
> > > > const&, void*, bool), void*, bool, int)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'ProfilerStart'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'ProfilerStop'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > > boost::cpp_regex_traits<char> > >::do_assign(char const*, char
> const*,
> > > > unsigned int)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> > > const*,
> > > > std::string>,
> > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> > > const*,
> > > > std::string> > >, boost::regex_traits<char,
> > > boost::cpp_regex_traits<char> >
> > > > >::find()'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> > > const*,
> > > > std::string>,
> > > > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> > > const*,
> > > > std::string> > >, boost::regex_traits<char,
> > > boost::cpp_regex_traits<char> >
> > > > >::construct_init(boost::basic_regex<char, boost::regex_traits<char,
> > > > boost::cpp_regex_traits<char> > > const&,
> > > > boost::regex_constants::_match_flags)'
> > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > os-ip.o):os-ip.c:function
> > > > ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or
> > > > `strerror_r' instead
> > > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > > os-ip.o):os-ip.c:function
> > > > ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror'
> or
> > > > `strerror_r' instead
> > > > collect2: error: ld returned 1 exit status
> > > > be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe for
> > target
> > > > 'be/build/debug/service/impalad' failed
> > > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > > CMakeFiles/Makefile2:3682: recipe for target
> > > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
> > > > CMakeFiles/Makefile2:3694: recipe for target
> > > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> > > >
> > >
> > >
> > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
>

Re: Error linking impalad with -build_shared_libs option

Posted by Laszlo Gaal <la...@cloudera.com>.
link.txt is generated by cmake based on the contents of CMakeLists.txt and
the project/environment settings. Most of the link-time dependencies seem
to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on settings
calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
$(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more into it
tomorrow.

Thanks,

    - LaszloG

On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <yo...@cloudera.com>
wrote:

> I observed the same linking error and digged this into further. The below
> is what I've found.
>
> *1. how to reproduce*
>
> # assuming that impala-config.sh is sourced
> $ cd ${IMPALA_HOME}
> *$ make -j8 impalad*
> ...
> [100%] Built target CodeGen
> Linking CXX executable ../../build/debug/service/impalad
> ...
> ../../build/debug/util/libUtil.so: undefined reference to
> `google_breakpad::ExceptionHandler::WriteMinidump(std::string const&, bool
> (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> ../../build/debug/util/libUtil.so: undefined reference to
> `snappy::MaxCompressedLength(unsigned long)'
> ../../build/debug/util/libUtil.so: undefined reference to `my_strlen'
> ../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'
> ../../build/debug/util/libUtil.so: undefined reference to
> `BZ2_bzDecompress'
> ...
>
>
> *$ cd ${IMPALA_HOME}/be/src/service*
> *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> ...
> ../../build/debug/util/libUtil.so: undefined reference to
> ...
> *../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'*
> *../../build/debug/util/libUtil.so: undefined reference to
> `BZ2_bzDecompress'*
> ../../build/debug/util/libUtil.so: undefined reference to
> *`snappy::RawCompress*(char const*,
> unsigned long, char*, unsigned long*)'
> ...
> boost::regex_traits<char, boost::cpp_regex_traits<char> > >::do_assign(char
> const*, char
> const*, unsigned int)'
> ...
> ../../build/debug/util/libUtil.so: undefined reference to *`snappy::*
> RawUncompress(char
> const*, unsigned long, char*)'
>
> *2. cause of the failure*
>
> $ cd ${IMPALA_HOME}/be/src/service
> *$ cat CMakeFiles/impalad.dir/link.txt*
> ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall -Wno-sign-compare
> -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> -Wno-deprecated ...
> *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> ../../../toolchain/lz4-svn/lib/liblz4.a*
> ...
> *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> ...
> *../../build/debug/util/libUtil.so*
> -lpthread -ldl -lssl -lcrypto
>
> Apparently, libUtil.so __should__ show before libsnappy.a and *regex.a.
>
>
> Next step is to find out who creates CMakeFiles/impalad.dir/link.txt and
> why libUtil.so is placed after libsnappy.a, liblz4.a, and libboost_regex.a
>
> Thank you,
> Yonghyun
>
>
>
>
>
> On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <he...@cloudera.com>
> wrote:
>
> > Only that it's IMPALA-3979. Does anyone have any theories yet?
> >
> > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > dtsirogiannis@cloudera.com> wrote:
> >
> > > Hey guys,
> > >
> > > When I build from the latest master using the -build_shared_libs
> option I
> > > get linking errors (see [1]). Building with static works fine. Any
> ideas?
> > >
> > > Thanks
> > > Dimitris
> > >
> > >
> > > [1]
> > > Linking CXX executable ../../build/debug/service/impalad
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzBuffToBuffCompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::MaxCompressedLength(unsigned long)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::RawCompress(char const*, unsigned long, char*, unsigned
> long*)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'LZ4_compress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzDecompressEnd'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzDecompressInit'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzBuffToBuffDecompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzDecompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::GetUncompressedLength(char const*, unsigned long, unsigned
> > > long*)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'LZ4_uncompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'my_strlen'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'google_breakpad::ExceptionHandler::WriteMinidump(std::string const&,
> > bool
> > > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > > breakpad::MinidumpDescriptor
> > > const&, bool (*)(void*), bool (*)(google_breakpad::MinidumpDescriptor
> > > const&, void*, bool), void*, bool, int)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'ProfilerStart'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'ProfilerStop'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'boost::basic_regex<char, boost::regex_traits<char,
> > > boost::cpp_regex_traits<char> > >::do_assign(char const*, char const*,
> > > unsigned int)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> > const*,
> > > std::string>,
> > > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> > const*,
> > > std::string> > >, boost::regex_traits<char,
> > boost::cpp_regex_traits<char> >
> > > >::find()'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> > const*,
> > > std::string>,
> > > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> > const*,
> > > std::string> > >, boost::regex_traits<char,
> > boost::cpp_regex_traits<char> >
> > > >::construct_init(boost::basic_regex<char, boost::regex_traits<char,
> > > boost::cpp_regex_traits<char> > > const&,
> > > boost::regex_constants::_match_flags)'
> > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > os-ip.o):os-ip.c:function
> > > ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or
> > > `strerror_r' instead
> > > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > > os-ip.o):os-ip.c:function
> > > ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or
> > > `strerror_r' instead
> > > collect2: error: ld returned 1 exit status
> > > be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe for
> target
> > > 'be/build/debug/service/impalad' failed
> > > make[3]: *** [be/build/debug/service/impalad] Error 1
> > > CMakeFiles/Makefile2:3682: recipe for target
> > > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
> > > CMakeFiles/Makefile2:3694: recipe for target
> > > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> > >
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>

Re: Error linking impalad with -build_shared_libs option

Posted by Yonghyun Hwang <yo...@cloudera.com>.
I observed the same linking error and digged this into further. The below
is what I've found.

*1. how to reproduce*

# assuming that impala-config.sh is sourced
$ cd ${IMPALA_HOME}
*$ make -j8 impalad*
...
[100%] Built target CodeGen
Linking CXX executable ../../build/debug/service/impalad
...
../../build/debug/util/libUtil.so: undefined reference to
`google_breakpad::ExceptionHandler::WriteMinidump(std::string const&, bool
(*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
../../build/debug/util/libUtil.so: undefined reference to
`snappy::MaxCompressedLength(unsigned long)'
../../build/debug/util/libUtil.so: undefined reference to `my_strlen'
../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'
../../build/debug/util/libUtil.so: undefined reference to `BZ2_bzDecompress'
...


*$ cd ${IMPALA_HOME}/be/src/service*
*$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
*cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
...
../../build/debug/util/libUtil.so: undefined reference to
...
*../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'*
*../../build/debug/util/libUtil.so: undefined reference to
`BZ2_bzDecompress'*
../../build/debug/util/libUtil.so: undefined reference to
*`snappy::RawCompress*(char const*,
unsigned long, char*, unsigned long*)'
...
boost::regex_traits<char, boost::cpp_regex_traits<char> > >::do_assign(char
const*, char
const*, unsigned int)'
...
../../build/debug/util/libUtil.so: undefined reference to *`snappy::*
RawUncompress(char
const*, unsigned long, char*)'

*2. cause of the failure*

$ cd ${IMPALA_HOME}/be/src/service
*$ cat CMakeFiles/impalad.dir/link.txt*
${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++    -Wall -Wno-sign-compare
-Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
-Wno-deprecated ...
*../../../toolchain/snappy-1.1.3/lib/libsnappy.a
../../../toolchain/lz4-svn/lib/liblz4.a*
...
*../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
...
*../../build/debug/util/libUtil.so*
-lpthread -ldl -lssl -lcrypto

Apparently, libUtil.so __should__ show before libsnappy.a and *regex.a.


Next step is to find out who creates CMakeFiles/impalad.dir/link.txt and
why libUtil.so is placed after libsnappy.a, liblz4.a, and libboost_regex.a

Thank you,
Yonghyun





On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson <he...@cloudera.com> wrote:

> Only that it's IMPALA-3979. Does anyone have any theories yet?
>
> On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> dtsirogiannis@cloudera.com> wrote:
>
> > Hey guys,
> >
> > When I build from the latest master using the -build_shared_libs option I
> > get linking errors (see [1]). Building with static works fine. Any ideas?
> >
> > Thanks
> > Dimitris
> >
> >
> > [1]
> > Linking CXX executable ../../build/debug/service/impalad
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'BZ2_bzBuffToBuffCompress'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'snappy::MaxCompressedLength(unsigned long)'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'LZ4_compress'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'BZ2_bzDecompressEnd'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'BZ2_bzDecompressInit'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'BZ2_bzBuffToBuffDecompress'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'BZ2_bzDecompress'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'snappy::GetUncompressedLength(char const*, unsigned long, unsigned
> > long*)'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'LZ4_uncompress'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'my_strlen'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'google_breakpad::ExceptionHandler::WriteMinidump(std::string const&,
> bool
> > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> > breakpad::MinidumpDescriptor
> > const&, bool (*)(void*), bool (*)(google_breakpad::MinidumpDescriptor
> > const&, void*, bool), void*, bool, int)'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'ProfilerStart'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'ProfilerStop'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'boost::basic_regex<char, boost::regex_traits<char,
> > boost::cpp_regex_traits<char> > >::do_assign(char const*, char const*,
> > unsigned int)'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> const*,
> > std::string>,
> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> const*,
> > std::string> > >, boost::regex_traits<char,
> boost::cpp_regex_traits<char> >
> > >::find()'
> > ../../build/debug/util/libUtil.so: error: undefined reference to
> > 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char
> const*,
> > std::string>,
> > std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char
> const*,
> > std::string> > >, boost::regex_traits<char,
> boost::cpp_regex_traits<char> >
> > >::construct_init(boost::basic_regex<char, boost::regex_traits<char,
> > boost::cpp_regex_traits<char> > > const&,
> > boost::regex_constants::_match_flags)'
> > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > os-ip.o):os-ip.c:function
> > ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or
> > `strerror_r' instead
> > /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> > os-ip.o):os-ip.c:function
> > ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or
> > `strerror_r' instead
> > collect2: error: ld returned 1 exit status
> > be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe for target
> > 'be/build/debug/service/impalad' failed
> > make[3]: *** [be/build/debug/service/impalad] Error 1
> > CMakeFiles/Makefile2:3682: recipe for target
> > 'be/src/service/CMakeFiles/impalad.dir/all' failed
> > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
> > CMakeFiles/Makefile2:3694: recipe for target
> > 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>

Re: Error linking impalad with -build_shared_libs option

Posted by Henry Robinson <he...@cloudera.com>.
Only that it's IMPALA-3979. Does anyone have any theories yet?

On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
dtsirogiannis@cloudera.com> wrote:

> Hey guys,
>
> When I build from the latest master using the -build_shared_libs option I
> get linking errors (see [1]). Building with static works fine. Any ideas?
>
> Thanks
> Dimitris
>
>
> [1]
> Linking CXX executable ../../build/debug/service/impalad
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'BZ2_bzBuffToBuffCompress'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'snappy::MaxCompressedLength(unsigned long)'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'LZ4_compress'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'BZ2_bzDecompressEnd'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'BZ2_bzDecompressInit'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'BZ2_bzBuffToBuffDecompress'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'BZ2_bzDecompress'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'snappy::GetUncompressedLength(char const*, unsigned long, unsigned
> long*)'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'snappy::RawUncompress(char const*, unsigned long, char*)'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'LZ4_uncompress'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'my_strlen'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'google_breakpad::ExceptionHandler::WriteMinidump(std::string const&, bool
> (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'google_breakpad::ExceptionHandler::ExceptionHandler(google_
> breakpad::MinidumpDescriptor
> const&, bool (*)(void*), bool (*)(google_breakpad::MinidumpDescriptor
> const&, void*, bool), void*, bool, int)'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'ProfilerStart'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'ProfilerStop'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'boost::basic_regex<char, boost::regex_traits<char,
> boost::cpp_regex_traits<char> > >::do_assign(char const*, char const*,
> unsigned int)'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char const*,
> std::string>,
> std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*,
> std::string> > >, boost::regex_traits<char, boost::cpp_regex_traits<char> >
> >::find()'
> ../../build/debug/util/libUtil.so: error: undefined reference to
> 'boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char const*,
> std::string>,
> std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*,
> std::string> > >, boost::regex_traits<char, boost::cpp_regex_traits<char> >
> >::construct_init(boost::basic_regex<char, boost::regex_traits<char,
> boost::cpp_regex_traits<char> > > const&,
> boost::regex_constants::_match_flags)'
> /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> os-ip.o):os-ip.c:function
> ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or
> `strerror_r' instead
> /opt/Impala-Toolchain/openldap-2.4.25/lib/libldap.a(
> os-ip.o):os-ip.c:function
> ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or
> `strerror_r' instead
> collect2: error: ld returned 1 exit status
> be/src/service/CMakeFiles/impalad.dir/build.make:166: recipe for target
> 'be/build/debug/service/impalad' failed
> make[3]: *** [be/build/debug/service/impalad] Error 1
> CMakeFiles/Makefile2:3682: recipe for target
> 'be/src/service/CMakeFiles/impalad.dir/all' failed
> make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
> CMakeFiles/Makefile2:3694: recipe for target
> 'be/src/service/CMakeFiles/impalad.dir/rule' failed
> make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
>



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679