[jira] [Commented] (BIGTOP-3308) Openssl fails to load in hadoop checknative -a on Debian 9

["Evans Ye (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059392#comment-17059392 ] 

Evans Ye commented on BIGTOP-3308:
----------------------------------

[~elukey] I use mac too and the core of Docker on Mac is still a hidden VM so I'm also experiencing the performance degradation as well.

Does 1.4 release strongly required by your org? We can discuss for that if there's a need. The reason we didn't do much backport n release in the past is there's no much user requirement sent in. From 1.4 RM point of view, we can do code release and skip those binary convenient artifacts. The effort should be much smaller than an entire release.

> Openssl fails to load in hadoop checknative -a on Debian 9
> ----------------------------------------------------------
>
>                 Key: BIGTOP-3308
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-3308
>             Project: Bigtop
>          Issue Type: Bug
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Luca Toscano
>            Assignee: Luca Toscano
>            Priority: Major
>             Fix For: 1.5.0, 1.4.1
>
>         Attachments: patch3-pom.diff
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Hi!
>  
> I am trying to switch from CDH 5.16 to BigTop 1.4. I am currently trying to upgrade on a test cluster with Debian 9.
> We use Spark 2.4.4 with RPC encryption+auth enabled, relying on openssl's libcrypto capabilities. I noticed that with BigTop 1.4 hadoop checknative reports an error for openssl:
> {code}
> openssl: false EVP_CIPHER_CTX_encrypting
> {code}
> On Debian 9 we have both openssl 1.10 and 1.0.2 deployed, so we configured libcrypto in this way:
> {code}
> 0 lrwxrwxrwx 1 root root 44 Feb 17 08:59 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
> {code}
> We had to add this symlink on CDH since a versioned libcrypto.so is not picked up, and libssl-dev on Debian 9 creates a symlink to libcrypto1.1.0, not 1.0.2 (that was failing).
> I see that EVP_CIPHER_CTX_encrypting seems guarded by a ifdef after https://issues.apache.org/jira/browse/BIGTOP-2932, so I guess that BigTop on Debian 9 builds against libssl1.1.0? (Still a bit ignorant, don't know how to check how things are build :()
> I tried to move the symlink to something like:
> {code}
> 0 lrwxrwxrwx 1 root root 42 Feb 17 10:31 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
> {code}
> And now checknative returns a different error:
> {code}
> openssl: false EVP_CIPHER_CTX_cleanup
> {code}
> The error in my opinion makes sense, since EVP_CIPHER_CTX_cleanup seems deprecated from openssl 1.1.0 onward. From the Hadoop mailing list I found this interesting thread, that shows the same problem:
> [http://mail-archives.us.apache.org/mod_mbox/hadoop-user/201910.mbox/%3cCALh-6sAug29Ua2iX+aZnR_TjzjQpwVBcVV_macQDxNYYgoOLzA@mail.gmail.com%3e]
> The main issue is that the reporter mentions libssl1.1.1, not 1.1.0, that should be working with the aforementioned patch. Looking at the code I can see that EVP_CIPHER_CTX_cleanup is not guarded by any ifdef (in [https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c]), so the error kinda makes sense to me.
>  
> Am I missing something obvious or is there a bug somewhere between Hadoop and BigTop?
>  
> Thanks in advance!
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)