[jira] [Resolved] (HDFS-16021) heap-use-after-free in hdfsThreadDestructor

["Akira Ajisaka (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Akira Ajisaka resolved HDFS-16021.
----------------------------------
    Resolution: Fixed

> heap-use-after-free in hdfsThreadDestructor
> -------------------------------------------
>
>                 Key: HDFS-16021
>                 URL: https://issues.apache.org/jira/browse/HDFS-16021
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.9.1, 2.9.2, 3.3.0
>            Reporter: Jeremy Coulon
>            Assignee: Akira Ajisaka
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.5.1, 3.6.0
>
>         Attachments: fix-hdfsThreadDestructor.patch, hdfs-asan.log
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Related to HDFS-12628 HDFS-13585 HDFS-14488 HDFS-15270 
>  
> We have experienced crashes located in libhdfs hdfsThreadDestructor() for a long time. Crash is almost systematic with OpenJ9 and more sporadic with Hotspot JVM.
>  
> I finally went to the root cause of this bug thanks to AddressSanitizer. This is quite difficult to setup because you need to rebuild both the test-case, hadoop and openjdk-hotspot >= 13 with specific compiler options.
>  
> See hdfs-asan.log for details.
>  
> *Analysis:*
> In hdfsThreadDestructor(), you are making several JNI calls in order to detach the thread from the JVM:
>  
> {code:java}
> /* Detach the current thread from the JVM */
> if (env) {
>   ret = (*env)->GetJavaVM(env, &vm);
>   /*
>    *  More code here...
>    */
> }{code}
> This is fine if the thread was created in the C/C++ world.
>  
> However if the thread was created in the Java world, this is absolutely wrong. When a Java thread terminates, the JVM deallocates some memory which contains (among other things) the thread specific JNIEnv. Then hdfsThreadDestructor() is called. The *env* variable is not NULL but points to memory which was just released. This is heap-use-after-free detected by ASan.
>  
> I have been working on a patch that fixes the issue (see attachment).
>  
> Here is the idea:
>  * In hdfsThreadDestructor(), we need to know if the thread was created by Java or C/C++ . If it was created by C/C++ we should make JNI calls in order to detach the current thread. If it was created by Java, we don't need to make any JNI call: thread is already detached.
>  * In getGlobalJNIEnv(), we can detect if the thread was created by Java or C/C++. It can be done by calling *vm->GetEnv()*. Then we store this information inside ThreadLocalState.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org