[jira] [Commented] (HDFS-16021) heap-use-after-free in hdfsThreadDestructor

["Jeremy Coulon (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17341967#comment-17341967 ] 

Jeremy Coulon commented on HDFS-16021:
--------------------------------------

Side-note:

I believe that HDFS-15270, was originally opened on OpenJ9 bug tracker by someone from my organisation. The proposed fix doesn't work!

 

Both *env* and **env* are invalid but not NULL.

> heap-use-after-free in hdfsThreadDestructor
> -------------------------------------------
>
>                 Key: HDFS-16021
>                 URL: https://issues.apache.org/jira/browse/HDFS-16021
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.9.1, 2.9.2, 3.3.0
>            Reporter: Jeremy Coulon
>            Priority: Major
>         Attachments: fix-hdfsThreadDestructor.patch, hdfs-asan.log
>
>
> Related to HDFS-12628 HDFS-13585 HDFS-14488 HDFS-15270 
>  
> We have experienced crashes located in libhdfs hdfsThreadDestructor() for a long time. Crash is almost systematic with OpenJ9 and more sporadic with Hotspot JVM.
>  
> I finally went to the root cause of this bug thanks to AddressSanitizer. This is quite difficult to setup because you need to rebuild both the test-case, hadoop and openjdk-hotspot >= 13 with specific compiler options.
>  
> See hdfs-asan.log for details.
>  
> *Analysis:*
> In hdfsThreadDestructor(), you are making several JNI calls in order to detach the thread from the JVM:
>  
> {code:java}
> /* Detach the current thread from the JVM */
> if (env) {
>   ret = (*env)->GetJavaVM(env, &vm);
>   /*
>    *  More code here...
>    */
> }{code}
> This is fine if the thread was created in the C/C++ world.
>  
> However if the thread was created in the Java world, this is absolutely wrong. When a Java thread terminates, the JVM deallocates some memory which contains (among other things) the thread specific JNIEnv. Then hdfsThreadDestructor() is called. The *env* variable is not NULL but points to memory which was just released. This is heap-use-after-free detected by ASan.
>  
> I have been working on a patch that fixes the issue (see attachment).
>  
> Here is the idea:
>  * In hdfsThreadDestructor(), we need to know if the thread was created by Java or C/C++ . If it was created by C/C++ we should make JNI calls in order to detach the current thread. If it was created by Java, we don't need to make any JNI call: thread is already detached.
>  * In getGlobalJNIEnv(), we can detect if the thread was created by Java or C/C++. It can be done by calling *vm->GetEnv()*. Then we store this information inside ThreadLocalState.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org