You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Luca Toscano (Jira)" <ji...@apache.org> on 2020/02/17 10:38:00 UTC
[jira] [Updated] (BIGTOP-3308) Openssl fails to load in hadoop
checknative -a on Debian 9
[ https://issues.apache.org/jira/browse/BIGTOP-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Toscano updated BIGTOP-3308:
---------------------------------
Description:
Hi!
I am trying to switch from CDH 5.16 to BigTop 1.4. I am currently trying to upgrade on a test cluster with Debian 9.
We use Spark 2.4.4 with RPC encryption+auth enabled, relying on openssl's libcrypto capabilities. I noticed that with BigTop 1.4 hadoop checknative reports an error for openssl:
openssl: false EVP_CIPHER_CTX_encrypting
On Debian 9 we have both openssl 1.10 and 1.0.2 deployed, so we configured libcrypto in this way:
0 lrwxrwxrwx 1 root root 44 Feb 17 08:59 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
We had to add this symlink on CDH since a versioned libcrypto.so is not picked up, and libssl-dev on Debian 9 creates a symlink to libcrypto1.1.0, not 1.0.2 (that was failing).
I see that EVP_CIPHER_CTX_encrypting seems guarded by a ifdef after https://issues.apache.org/jira/browse/BIGTOP-2932, so I guess that BigTop on Debian 9 builds against libssl1.1.0? (Still a bit ignorant, don't know how to check how things are build :()
I tried to move the symlink to something like:
0 lrwxrwxrwx 1 root root 42 Feb 17 10:31 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
And now checknative returns a different error:
openssl: false EVP_CIPHER_CTX_cleanup
The error in my opinion makes sense, since EVP_CIPHER_CTX_cleanup seems deprecated from openssl 1.1.0 onward. From the Hadoop mailing list I found this interesting thread, that shows the same problem:
[http://mail-archives.us.apache.org/mod_mbox/hadoop-user/201910.mbox/%3cCALh-6sAug29Ua2iX+aZnR_TjzjQpwVBcVV_macQDxNYYgoOLzA@mail.gmail.com%3e]
The main issue is that the reporter mentions libssl1.1.1, not 1.1.0, that should be working with the aforementioned patch. Looking at the code I can see that EVP_CIPHER_CTX_cleanup is not guarded by any ifdef (in [https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c]), so the error kinda makes sense to me.
Am I missing something obvious or is there a bug somewhere between Hadoop and BigTop?
Thanks in advance!
was:
Hi!
I am trying to switch from CDH 5.16 to BigTop 1.4. I am currently trying to upgrade on a test cluster with Debian 9.
We use Spark 2.4.4 with RPC encryption+auth enabled, relying on openssl's libcrypto capabilities. I noticed that with BigTop 1.4 hadoop checknative reports an error for openssl:
openssl: false EVP_CIPHER_CTX_encrypting
Since on Debian 9 we have both openssl 1.10 and 1.0.2 deployed, we configured libcrypto in this way:
0 lrwxrwxrwx 1 root root 44 Feb 17 08:59 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
We had to add this symlink on CDH since a versioned libcrypto.so is not picked up, and libssl-dev on Debian 9 creates a symlink to libcrypto1.1.0, not 1.0.2 (that was failing).
I see that EVP_CIPHER_CTX_encrypting seems guarded by a ifdef after https://issues.apache.org/jira/browse/BIGTOP-2932, so I guess that BigTop on Debian 9 builds against libssl1.1.0? (Still a bit ignorant, don't know how to check how things are build :()
I tried to move the symlink to something like:
0 lrwxrwxrwx 1 root root 42 Feb 17 10:31 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
And now checknative returns a different error:
openssl: false EVP_CIPHER_CTX_cleanup
The error in my opinion makes sense, since EVP_CIPHER_CTX_cleanup seems deprecated from openssl 1.1.0 onward. From the Hadoop mailing list I found this interesting thread, that shows the same problem:
[http://mail-archives.us.apache.org/mod_mbox/hadoop-user/201910.mbox/%3cCALh-6sAug29Ua2iX+aZnR_TjzjQpwVBcVV_macQDxNYYgoOLzA@mail.gmail.com%3e]
The main issue is that the reporter mentions libssl1.1.1, not 1.1.0, that should be working with the aforementioned patch. Looking at the code I can see that EVP_CIPHER_CTX_cleanup is not guarded by any ifdef (in [https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c]), so the error kinda makes sense to me.
Am I missing something obvious or is there a bug somewhere between Hadoop and BigTop?
Thanks in advance!
> Openssl fails to load in hadoop checknative -a on Debian 9
> ----------------------------------------------------------
>
> Key: BIGTOP-3308
> URL: https://issues.apache.org/jira/browse/BIGTOP-3308
> Project: Bigtop
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Luca Toscano
> Priority: Major
>
> Hi!
>
> I am trying to switch from CDH 5.16 to BigTop 1.4. I am currently trying to upgrade on a test cluster with Debian 9.
> We use Spark 2.4.4 with RPC encryption+auth enabled, relying on openssl's libcrypto capabilities. I noticed that with BigTop 1.4 hadoop checknative reports an error for openssl:
>
> openssl: false EVP_CIPHER_CTX_encrypting
>
> On Debian 9 we have both openssl 1.10 and 1.0.2 deployed, so we configured libcrypto in this way:
>
> 0 lrwxrwxrwx 1 root root 44 Feb 17 08:59 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
>
> We had to add this symlink on CDH since a versioned libcrypto.so is not picked up, and libssl-dev on Debian 9 creates a symlink to libcrypto1.1.0, not 1.0.2 (that was failing).
> I see that EVP_CIPHER_CTX_encrypting seems guarded by a ifdef after https://issues.apache.org/jira/browse/BIGTOP-2932, so I guess that BigTop on Debian 9 builds against libssl1.1.0? (Still a bit ignorant, don't know how to check how things are build :()
>
> I tried to move the symlink to something like:
> 0 lrwxrwxrwx 1 root root 42 Feb 17 10:31 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>
> And now checknative returns a different error:
>
> openssl: false EVP_CIPHER_CTX_cleanup
>
> The error in my opinion makes sense, since EVP_CIPHER_CTX_cleanup seems deprecated from openssl 1.1.0 onward. From the Hadoop mailing list I found this interesting thread, that shows the same problem:
>
> [http://mail-archives.us.apache.org/mod_mbox/hadoop-user/201910.mbox/%3cCALh-6sAug29Ua2iX+aZnR_TjzjQpwVBcVV_macQDxNYYgoOLzA@mail.gmail.com%3e]
>
> The main issue is that the reporter mentions libssl1.1.1, not 1.1.0, that should be working with the aforementioned patch. Looking at the code I can see that EVP_CIPHER_CTX_cleanup is not guarded by any ifdef (in [https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c]), so the error kinda makes sense to me.
>
> Am I missing something obvious or is there a bug somewhere between Hadoop and BigTop?
>
> Thanks in advance!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)