You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2024/01/02 11:58:00 UTC

[jira] [Commented] (MAPREDUCE-7467) in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801755#comment-17801755 ] 

Steve Loughran commented on MAPREDUCE-7467:
-------------------------------------------

pluggable would be nice but complex; we still support java 8, though I hope we can move trunk on from there, after which we can move completely.

Have you played with using open SSL as your https support, it is a lot faster than a bit brittle.

> in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-7467
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7467
>             Project: Hadoop Map/Reduce
>          Issue Type: Wish
>    Affects Versions: 3.3.5, 3.3.3, 3.3.4, 3.3.6
>            Reporter: Arnaud Nauwynck
>            Priority: Critical
>
> As described in Jira Title: "in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive"
> Few remarks:
> 1/ The official Azure SDK supports either OkHttp or Netty for the Http transport.
> 2/ the actual hadoop-azure use the class java.net.HttpURLConnection, which is slow.
>   It does not use Http2, does not optimize SSL Hand-shake very well, and does not keep TCP connection alive for re-use.
> 3/ JDK since version >=11 have a new class HttpClient which should be a better replacement 
> 4/ it might be possible to introduce a configuration property (with defaut to use legacy class) , and an abstract factory to create connection via either HttpURLConnection or any other pluggeable implementation (jdk 11 HttpClient, OkHttp, Netty, ...)
> 5/ the official Azure SDK is maintained by Microsoft, so should better follow bug fixes and improvements than custom hadoop implementation?
> [https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable
> |https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable]
> 6/ when we use code with the official Azure SDK and Hadoop(in Spark), it is chocking to have 2 different implementations within the same JVM... 
> 7/ The official Azure SDK has more features that what allows the legacy hadoop class FileSystem to do... In particular, we can append (=upload) file by multiple threads (upload by fragments at different offsets), then flush when every fragments are sent.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org