You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jaehyun Kim (Jira)" <ji...@apache.org> on 2025/08/26 05:45:00 UTC

[jira] [Created] (FLINK-38284) Prepare to upgrade hadoop version to 3.4.2 across Flink's Hadoop-based FS connectors for OpenSSL 3 and Java 17 compatibility

Jaehyun Kim created FLINK-38284:
-----------------------------------

             Summary: Prepare to upgrade hadoop version to 3.4.2 across Flink's Hadoop-based FS connectors for OpenSSL 3 and Java 17 compatibility
                 Key: FLINK-38284
                 URL: https://issues.apache.org/jira/browse/FLINK-38284
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / FileSystem, FileSystems
            Reporter: Jaehyun Kim


h3. Description

Apache Hadoop has merged [PR #7032|https://github.com/apache/hadoop/pull/7032] and HADOOP-19262, upgrading wildfly-openssl to 2.1.6.Final to compatibility with Java 17 and OpenSSL 3. This fix is planned to be included in the upcoming Hadoop 3.4.2 release.

Currently, Flink sets in {{flink-fliesystems/pom.xml}} :
{code:java}
<fs.hadoopshaded.version>3.3.4</fs.hadoopshaded.version> {code}
which means modules like {{flink-azure-fs-hadoop.jar}} transitively include {{wildfly-oepnssl:1.0.7:Final}} via {{{}hadoop-azure:3.3.4{}}}. This version is not compatible with OpenSSL 3 and causes runtime issues on modern platforms.
h3. Impact and Scope

This issue originates in Apache Hadoop's {{hadoop-azure}} module, which transitively includes an outdated version of {{{}wildfly-openssl{}}}. As a result, all Flink modules depending on this (e.g., {{{}flink-azure-fs-hadoop{}}}) are affected.

Furthermore, other Flink filesystem connectors that rely on Hadoop (directly or via {{{}flink-shaded-hadoop{}}}) may also benefit from this upgrade:
 * {{flink-azure-fs-hadoop}}

 * {{flink-gs-fs-hadoop}}

 * {{{}flink-oss-fs-hadoop{}}}{{{}{}}}
 * {{{}flink-s3-fs-hadoop{}}}{{{}{}}}

This change is particularly relevant for users running Flink on:
 * {*}Java 17{*}, where {{X509V1CertImpl}} was removed from the JDK

 * *OpenSSL 3.x systems* (e.g., RHEL 9), where older {{wildfly-openssl}} versions fail to load

h3. Motivation

Upgrading to {{hadoop-azure:3.4.2}} will:
 * Ensure compatibility with Java 17+ and OpenSSL 3

 * Resolve {{ClassNotFoundException: com.sun.security.cert.internal.x509.X509V1CertImpl}} errors on OpenSSL 1.1-based systems (e.g., RHEL 8.10)
 * Align with Hadoop upstream fixes

 * Avoid of performance-impacting workarounds like forcing {{fs.azure.ssl.channel.mode=Default_JSSE}}

 * Even when JSSE fallback avoids the crash, {*}it is not ideal for performance and stability{*}.
Using native OpenSSL via JNI (as intended by {{{}wildfly-openssl{}}}) is preferred in high-throughput or secure production environments.

h3. Proposed Plan

Once Hadoop 3.4.2 is officially released:
 # Update {{fs.hadoopshaded.version}} to {{3.4.2}} in {{flink-filesystems/pom.xml}} 

 #  
 # Verify and update NOTICE/LICENSE files as required

 # Rebuild {{flink-azure-fs-hadoop}} to confirm correct shading of the updated dependencies

 # Ensure that native SSL initialization works in both OpenSSL 1.1 and 3 environments

 # Optionally, update test coverage for ABFS + SSL

This ticket serves to track the upgrade preparation and corresponding work once the upstream Hadoop release is available.
h3. Environment Affected
 * Flink 1.19.0 - 2.1.0

 * Java 17 (OracleJDK, OpenJDK, Amazon Corretto)

 * RHEL 8.10 (OpenSSL 1.1.1) → native loads, causes error
{code:java}
[ERROR] org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error occurred in the cluster entrypoint.java.util.concurrent.CompletionException: java.lang.RuntimeException: java.lang.IllegalStateException: javax.security.cert.CertificateException: Could not find class: java.lang.ClassNotFoundException: com/sun/security/cert/internal/x509/X509V1CertImpl{code}
 

 * RHEL 9.3 (OpenSSL 3.x) → native fails, JSSE fallback
{code:java}
[DEBUG] org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory   [] - Failed to load OpenSSL. Falling back to the JSSE{code}
 

 * ABFS with HA enabled ({{{}abfss://{}}})

h3. Workarounds Today
 * Set {{fs.azure.ssl.channel.mode=Default_JSSE}} to disable native OpenSSL

 * Avoid OpenSSL 1.1 platforms
 * Remove the {{wildfly-openssl}} JAR from the opt plugin (not ideal)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)