You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/06/15 23:43:00 UTC
[jira] [Commented] (IMPALA-9804) Fix up LD_LIBRARY_PATH for
bin/impala-shell.sh
[ https://issues.apache.org/jira/browse/IMPALA-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136205#comment-17136205 ]
ASF subversion and git services commented on IMPALA-9804:
---------------------------------------------------------
Commit 13fbe510c0d70a8cbe82f0ca83f59b3faf5353c8 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=13fbe51 ]
IMPALA-9838: Switch to GCC 7.5.0
This upgrades GCC and libstdc++ to version 7.5.0. There
have been ABI changes since 4.9.2, so this means that
the native-toolchain produced with the new compiler is
not interoperable with one produced by the old compiler.
To allow that transition, IMPALA_TOOLCHAIN_PACKAGES_HOME
is now a subdirectory of IMPALA_TOOLCHAIN
(toolchain-packages-gcc${IMPALA_GCC_VERSION}) to distinguish
it from the old packages.
Some Python packages in the impala-python virtualenv are
compiled using the toolchain GCC and now use the new ABI.
This leads to two changes:
1. When constructing the LD_LIBRARY_PATH for impala-python,
we include the GCC libstdc++ libraries. Otherwise, certain
Python packages that use C++ fail on older OSes like Centos 7.
This fixes IMPALA-9804.
2. Since developers work on various branches, this changes
the virtualenv's directory location to a directory with
the GCC version in the name. This allows the virtualenv
built with GCC 7 to coexist with the current virtualenv
built with GCC 4.9.2. The location for the old virtualenv is
${IMPALA_HOME}/infra/python/env. The new location is
${IMPALA_HOME}/infra/python/env-gcc${IMPALA_GCC_VERSION}. This
required updating several impala-python scripts.
There are various odds-and-ends related to the transition:
1. Due to the small string optimization, the size of std::string
changed, which means that various data structures also changed
in size. This required updating some static asserts.
2. There is a bug in clang-tidy that reports a use-after-free
for some code using std::shared_ptr. Clang is not modeling
the shared_ptr correctly, so it is a false-positive. As a workaround,
this disables the clang-analyzer-cplusplus.NewDelete diagnostic.
3. Various small compilation fixes (includes, etc).
Performance testing:
- Ran single-node performance tests on TPC-H for the following
configurations:
- TPC-H Parquet scale 30 with normal configurations
- TPC-H Parquet scale 30 with codegen disabled
- TPC-H Kudu scale 10
None found any significant regressions. Full results are
posted on the JIRA.
- Ran single-node performance tests on targeted-perf scale 10.
No significant regressions.
- The size of binaries (impalad, etc) is slightly smaller with the new GCC:
GCC 4.9.2 release impalad binary: 545664
GCC 7.5.0 release impalad binary: 539900
- Compilation in DEBUG mode is roughly 15-25% faster
Functional testing:
- Ran core jobs, exhaustive release jobs, UBSAN
Change-Id: Ia0beb2b618ba669c9699f8dbc0c52d1203d004e4
Reviewed-on: http://gerrit.cloudera.org:8080/16045
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Fix up LD_LIBRARY_PATH for bin/impala-shell.sh
> ----------------------------------------------
>
> Key: IMPALA-9804
> URL: https://issues.apache.org/jira/browse/IMPALA-9804
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> When running with GCC 7.5.0 on Centos 7, dataload hits the following error:
> {noformat}
> 23:51:41 Caching test tables (logging to /data/jenkins/workspace/impala-private-parameterized/repos/Impala/logs/data_loading/cache-test-tables.log)...
> 23:51:42 FAILED (Took: 0 min 1 sec)
> 23:51:42 'cache-test-tables' failed. Tail of log:
> 23:51:42 Log for command 'cache-test-tables'
> 23:51:42 CACHING tpch.nation AND functional.alltypestiny
> 23:51:42 Traceback (most recent call last):
> 23:51:42 File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/shell/impala_shell.py", line 42, in <module>
> 23:51:42 from impala_client import ImpalaHS2Client, ImpalaBeeswaxClient, QueryOptionLevels
> 23:51:42 File "/data0/jenkins/workspace/impala-private-parameterized/repos/Impala/shell/impala_client.py", line 27, in <module>
> 23:51:42 import sasl
> 23:51:42 File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/infra/python/env/lib/python2.7/site-packages/sasl/__init__.py", line 15, in <module>
> 23:51:42 from sasl.saslwrapper import *
> 23:51:42 ImportError: /data/jenkins/workspace/impala-private-parameterized/repos/Impala/infra/python/env/lib/python2.7/site-packages/sasl/saslwrapper.so: undefined symbol: _ZTVNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE{noformat}
> This command is using bin/impala-shell.sh, which uses impala-python to run the shell. The python packages for the infra/python/env virtualenv are compiled using the toolchain GCC. Apparently, at least one of them (sasl) is compiling c++ code, so this means that the compiled python packages need to be able to find the appropriate libstdc++. Centos 7 has an older libstdc++, so the system libstdc++ can't satisfy these symbols.
> This seems like it should be limited to impala-python, and so it may be enough to fix only bin/impala-shell.sh. One fix is to add the GCC paths to the paths returned by "infra/python/bootstrap_virtualenv.py --print-ld-library-path"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org