You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2022/10/04 17:05:00 UTC
[jira] [Commented] (KUDU-3404) glog 0.6.0 increases the TLS usage of libkudu_client.so substantially
[ https://issues.apache.org/jira/browse/KUDU-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612691#comment-17612691 ]
Michael Smith commented on KUDU-3404:
-------------------------------------
Is this something it would make sense to configure only in Impala's build of Kudu?
> glog 0.6.0 increases the TLS usage of libkudu_client.so substantially
> ---------------------------------------------------------------------
>
> Key: KUDU-3404
> URL: https://issues.apache.org/jira/browse/KUDU-3404
> Project: Kudu
> Issue Type: Bug
> Components: client
> Affects Versions: 1.17.0
> Reporter: Joe McDonnell
> Priority: Critical
> Attachments: 0001-Add-WITH_TLS-OFF-to-glog-build-definition.patch
>
>
> Glog 0.4 introduced supported for using thread local storage for its buffer. This feature is controlled by the WITH_TLS CMake variable, and it defaults to ON. See [https://github.com/google/glog/commit/2df0ca34aa3000dadf76633ca700abf0bf50756d] . When Kudu upgraded to glog 0.6.0 as part of the M1 fixes in "[KUDU-3374 Add support for M1 and macOS Monterey|https://github.com/apache/kudu/commit/543e128d473f8f7836e605bba8cd6512fa918550]", it increased the thread local storage usage by >30000 bytes.
> {noformat}
> # Older libkudu_client.so has 0x100 = 256 bytes of TLS:
> $ readelf -l libkudu_client.so | grep "TLS" -A1
> TLS 0x00000000007d14c0 0x00000000007d24c0 0x00000000007d24c0
> 0x0000000000000080 0x0000000000000100 R 0x40
> # Newer libkudu_client.so has 0x77b9 = 30649 bytes of TLS:
> $ readelf -l libkudu_client.so.0 | grep TLS -A1
> TLS 0x0000000000751280 0x0000000000752280 0x0000000000752280
> 0x0000000000000080 0x00000000000077b9 R 40{noformat}
> This is a problem for Impala, because Impala starts a JVM. There are certain JVM threads (like the "reaper thread") that have very small stacks (e.g. 32KB) and with glibc the TLS space is allocated at the expense of stack space. 30k of TLS usage leaves very little for the reaper thread. There are a series of bugs where the Java reaper thread hits a StackOverflowException because of high TLS usage. This can cause various symptoms including hangs.
> GLIBC message thread: [https://sourceware.org/bugzilla/show_bug.cgi?id=11787]
> JDK bugs: [JDK-8217475|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8217475], [JDK-8225035|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8225035]
> To resolve Impala's problem, it would be useful to build libkudu_client.so with glog's WITH_TLS=OFF.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)