You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/12/23 16:54:17 UTC
[GitHub] [hadoop] packet23 opened a new pull request, #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
packet23 opened a new pull request, #5256:
URL: https://github.com/apache/hadoop/pull/5256
### Description of PR
OpenSSL 3.x broke existing ABI - the symbols for
* `EVP_CIPHER_CTX_block_size` and
* `EVP_CIPHER_CTX_encrypting`
are now called
* `EVP_CIPHER_CTX_get_block_size` and
* `EVP_CIPHER_CTX_is_encrypting` respectively (c.f. commit [ed576acdf591d4164905ab98e89ca5a3b99d90ab](https://github.com/openssl/openssl/commit/ed576acdf591d4164905ab98e89ca5a3b99d90ab) that landed in [openssl-3.0.0-beta1](https://github.com/openssl/openssl/releases/tag/openssl-3.0.0-beta1))
The PR changes the hadoop-common native code such that when compiled against OpenSSL 3.x, it can successfully load the OpenSSL 3.x symbols at runtime.
### How was this patch tested?
A patched version of Hadoop 3.3.4 was compiled on an x86-64 Ubuntu 22.04 machine and `hadoop checknative` was invoked.
### For code changes:
- [ X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1385917648
thanks for this. wildfly is a source of ongoing pain and pretty brittle across openssl versions. probably a full update is needed there.
givent eh other places where things fail, is checknative correct in reporting incompatibility? as at least it is a fail-fast test for all of this.
which means: fixing it should be the last action.
* why not create a "support openssl 3.x" JIRA with the checknative one moved to being a subtask.
* we can discuss whether it is time to retire hadoop-pipes
* yarn issue is for a YARN jira
* opensslSecureRandom should be included in the checknative and we could make it probeable (i.e save stack on init failure to an accessible field; checknative could verify this is empty)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1364550085
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 3s | | trunk passed |
| +1 :green_heart: | compile | 20m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 81m 25s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 20m 3s | | the patch passed |
| +1 :green_heart: | cc | 20m 3s | | the patch passed |
| +1 :green_heart: | golang | 20m 3s | | the patch passed |
| +1 :green_heart: | javac | 20m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 23s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 18m 43s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not generate ASF License warnings. |
| | | 146m 30s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec |
| | hadoop.crypto.TestCryptoCodec |
| | hadoop.service.launcher.TestServiceInterruptHandling |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5256 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux 066f4e98dd4a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5c9b44f6e7a204cb5831c171f486cafc1d53e96 |
| Default Java | Red Hat, Inc.-1.8.0_352-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/2/testReport/ |
| Max. process+thread count | 1301 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#discussion_r1056819287
##########
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
##########
@@ -35,8 +35,14 @@ static void (*dlsym_EVP_CIPHER_CTX_init)(EVP_CIPHER_CTX *);
#endif
static int (*dlsym_EVP_CIPHER_CTX_set_padding)(EVP_CIPHER_CTX *, int);
static int (*dlsym_EVP_CIPHER_CTX_test_flags)(const EVP_CIPHER_CTX *, int);
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
Review Comment:
1. how about adding a comment to say "the names were changed so the probes have to change their type.
2. do the actual typedefs need to be made exclusive? can see that if the actual typedef name was being recycled, but here they are being given new names
##########
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
##########
@@ -207,10 +233,20 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_test_flags, \
dlsym_EVP_CIPHER_CTX_test_flags, env, \
openssl, "EVP_CIPHER_CTX_test_flags");
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
Review Comment:
these are all static...the probes need to be compiled for openssl 2 vs 3. do we need this, or can it just look for either sets of symbols and be happy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] packet23 commented on a diff in pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
packet23 commented on code in PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#discussion_r1056843734
##########
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
##########
@@ -35,8 +35,14 @@ static void (*dlsym_EVP_CIPHER_CTX_init)(EVP_CIPHER_CTX *);
#endif
static int (*dlsym_EVP_CIPHER_CTX_set_padding)(EVP_CIPHER_CTX *, int);
static int (*dlsym_EVP_CIPHER_CTX_test_flags)(const EVP_CIPHER_CTX *, int);
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
Review Comment:
addressed 2. in 6c17fcfe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1364278630
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 35m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 11s | | trunk passed |
| +1 :green_heart: | compile | 20m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 81m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 19m 57s | | the patch passed |
| +1 :green_heart: | cc | 19m 57s | | the patch passed |
| +1 :green_heart: | golang | 19m 57s | | the patch passed |
| +1 :green_heart: | javac | 19m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 11s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 18m 29s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not generate ASF License warnings. |
| | | 181m 7s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec |
| | hadoop.crypto.TestCryptoCodec |
| | hadoop.service.launcher.TestServiceInterruptHandling |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5256 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux 76e9e4ea09ff 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4ce0602f3fdfd9cc31005763eebdc6daf95fd9fe |
| Default Java | Red Hat, Inc.-1.8.0_352-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/1/testReport/ |
| Max. process+thread count | 2661 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/1/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1364568395
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 17s | | trunk passed |
| +1 :green_heart: | compile | 20m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 81m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 20m 6s | | the patch passed |
| +1 :green_heart: | cc | 20m 7s | | the patch passed |
| +1 :green_heart: | golang | 20m 6s | | the patch passed |
| +1 :green_heart: | javac | 20m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 1s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 18m 33s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not generate ASF License warnings. |
| | | 145m 26s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec |
| | hadoop.crypto.TestCryptoCodec |
| | hadoop.service.launcher.TestServiceInterruptHandling |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5256 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux dca52a0e3810 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6c17fcfeb75746078b72cc05913b23ff499fc74f |
| Default Java | Red Hat, Inc.-1.8.0_352-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/3/testReport/ |
| Max. process+thread count | 1669 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5256/3/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] packet23 commented on a diff in pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
packet23 commented on code in PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#discussion_r1056827158
##########
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
##########
@@ -207,10 +233,20 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_test_flags, \
dlsym_EVP_CIPHER_CTX_test_flags, env, \
openssl, "EVP_CIPHER_CTX_test_flags");
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
Review Comment:
Not quite sure - let me check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] packet23 commented on pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
packet23 commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1366040812
After digging a bit a more into Hadoop's usage of OpenSSL, I believe the PR fixes `hadoop checknative`, but there are other places that are not covered by `hadoop checknative`:
* `org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory` - uses OpenSSL via wildfly-openssl 1.0.7.Final (wildfly-openssl >= 2.2.0.Final supports OpenSSL 3.x), not covered by `hadoop checknative`
* `org.apache.hadoop.crypto.random.OpensslSecureRandom` - not covered by `hadoop checknative` as it seems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] t8m commented on pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
t8m commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1365756378
OpenSSL 1.1.1 is not ABI compatible (and only partially API compatible) with OpenSSL 3.x.
All the 3.x versions should be upwards ABI compatible - i.e. something compiled against 3.x will work with 3.(x+1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] packet23 commented on a diff in pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
packet23 commented on code in PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#discussion_r1056854431
##########
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
##########
@@ -207,10 +233,20 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_test_flags, \
dlsym_EVP_CIPHER_CTX_test_flags, env, \
openssl, "EVP_CIPHER_CTX_test_flags");
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
Review Comment:
I need some more information here:
1. With "look for either sets of symbols" do you mean moving the conditional symbol loading from compile time to runtime?
2. What contract do we have for native code regarding OpenSSL ABIs?
The PR delivers the following contract:
* If compiled against OpenSSL X.Y API, it will only work at runtime if OpenSSL X.Y ABI is available.
To my knowledge, this is not a regression, looking at the other conditional symbol loads and their usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] packet23 commented on a diff in pull request #5256: HADOOP-18583. Fix loading of OpenSSL 3.x symbols
Posted by GitBox <gi...@apache.org>.
packet23 commented on code in PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#discussion_r1056825019
##########
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
##########
@@ -35,8 +35,14 @@ static void (*dlsym_EVP_CIPHER_CTX_init)(EVP_CIPHER_CTX *);
#endif
static int (*dlsym_EVP_CIPHER_CTX_set_padding)(EVP_CIPHER_CTX *, int);
static int (*dlsym_EVP_CIPHER_CTX_test_flags)(const EVP_CIPHER_CTX *, int);
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
Review Comment:
1. addressed in f5c9b44f
2. wasn't sure which name pops up in `hadoop checknative`in case hadoop-common was compiled against OpenSSL 3.x and run with OpenSSL 1.x. Given that the symbols have same signature, the patch could be restricted to only `Java_org_apache_hadoop_crypto_OpensslCipher_initIDs`. Let me verify that...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-18583. Fix loading of OpenSSL 3.x symbols [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1977458743
> hadoop-yarn-server-nodemanager's container-executor
no idea. best to discuss on yarn-dev
regarding crypton and checknative, we may to work on that. IMO checknative should look for openssl, at least if we add a -openssl argument.
I don't think we have specific openssl issues. In #6425 I had to add error string matching for openssl 1 messages indicating stale https connections (these were surfacing deep in the AWS error stack). For 3.x it'd be good to know that these strings were the same -or update them. Maybe you can create an uber-jira "support openssl 3"
one more thing: what is the openssl FIPS story? as for strict fips support we don't just want to talk to fips endpoints, we want to run on hosts where the ssl lib doesn't have the untrusted algorithms at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-18583. Fix loading of OpenSSL 3.x symbols [hadoop]
Posted by "jasonwzs (via GitHub)" <gi...@apache.org>.
jasonwzs commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1968123368
@packet23 @steveloughran , can you provide more details on the individual issues related to openssl 3 support? It would be nice if we create jira issues for tracking purpose.
I am more interested in the yarn issue mentioned for hadoop-yarn-server-nodemanager's container-executor. Where is the related code and what is the impact?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org