You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2024/01/02 11:58:00 UTC
[jira] [Commented] (MAPREDUCE-7467) in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801755#comment-17801755 ]
Steve Loughran commented on MAPREDUCE-7467:
-------------------------------------------
pluggable would be nice but complex; we still support java 8, though I hope we can move trunk on from there, after which we can move completely.
Have you played with using open SSL as your https support, it is a lot faster than a bit brittle.
> in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7467
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7467
> Project: Hadoop Map/Reduce
> Issue Type: Wish
> Affects Versions: 3.3.5, 3.3.3, 3.3.4, 3.3.6
> Reporter: Arnaud Nauwynck
> Priority: Critical
>
> As described in Jira Title: "in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive"
> Few remarks:
> 1/ The official Azure SDK supports either OkHttp or Netty for the Http transport.
> 2/ the actual hadoop-azure use the class java.net.HttpURLConnection, which is slow.
> It does not use Http2, does not optimize SSL Hand-shake very well, and does not keep TCP connection alive for re-use.
> 3/ JDK since version >=11 have a new class HttpClient which should be a better replacement
> 4/ it might be possible to introduce a configuration property (with defaut to use legacy class) , and an abstract factory to create connection via either HttpURLConnection or any other pluggeable implementation (jdk 11 HttpClient, OkHttp, Netty, ...)
> 5/ the official Azure SDK is maintained by Microsoft, so should better follow bug fixes and improvements than custom hadoop implementation?
> [https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable
> |https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable]
> 6/ when we use code with the official Azure SDK and Hadoop(in Spark), it is chocking to have 2 different implementations within the same JVM...
> 7/ The official Azure SDK has more features that what allows the legacy hadoop class FileSystem to do... In particular, we can append (=upload) file by multiple threads (upload by fragments at different offsets), then flush when every fragments are sent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org