[jira] [Created] (KUDU-3404) glog 0.6.0 increases the TLS usage of libkudu_client.so substantially

["Joe McDonnell (Jira)" <ji...@apache.org>]

Joe McDonnell created KUDU-3404:
-----------------------------------

             Summary: glog 0.6.0 increases the TLS usage of libkudu_client.so substantially
                 Key: KUDU-3404
                 URL: https://issues.apache.org/jira/browse/KUDU-3404
             Project: Kudu
          Issue Type: Bug
          Components: client
    Affects Versions: 1.17.0
            Reporter: Joe McDonnell


Glog 0.4 introduced supported for using thread local storage for its buffer. This feature is controlled by the WITH_TLS CMake variable, and it defaults to ON. See [https://github.com/google/glog/commit/2df0ca34aa3000dadf76633ca700abf0bf50756d] . When Kudu upgraded to glog 0.6.0 as part of the M1 fixes in "[KUDU-3374 Add support for M1 and macOS Monterey|https://github.com/apache/kudu/commit/543e128d473f8f7836e605bba8cd6512fa918550]", it increased the thread local storage usage by >30000 bytes.
{noformat}
# Older libkudu_client.so has 0x100 = 256 bytes of TLS:
$ readelf -l libkudu_client.so | grep "TLS" -A1
  TLS            0x00000000007d14c0 0x00000000007d24c0 0x00000000007d24c0
                 0x0000000000000080 0x0000000000000100  R      0x40

# Newer libkudu_client.so has 0x77b9 = 30649 bytes of TLS:
$ readelf -l libkudu_client.so.0 | grep TLS -A1
  TLS            0x0000000000751280 0x0000000000752280 0x0000000000752280
                 0x0000000000000080 0x00000000000077b9  R      40{noformat}
This is a problem for Impala, because Impala starts a JVM. There are certain JVM threads (like the "reaper thread") that have very small stacks (e.g. 32KB) and with glibc the TLS space is allocated at the expense of stack space. 30k of TLS usage leaves very little for the reaper thread. There are a series of bugs where the Java reaper thread hits a StackOverflowException because of high TLS usage. This can cause various symptoms including hangs.

GLIBC message thread: [https://sourceware.org/bugzilla/show_bug.cgi?id=11787]

JDK bugs:  [JDK-8217475|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8217475], [JDK-8225035|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8225035]

To resolve Impala's problem, it would be useful to build libkudu_client.so with glog's WITH_TLS=OFF.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)