You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2022/03/14 12:40:00 UTC
[jira] [Commented] (SPARK-28254) Use More Generic Netty Interfaces to Support Fabric Network in Spark
[ https://issues.apache.org/jira/browse/SPARK-28254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17506220#comment-17506220 ]
Yang Jie commented on SPARK-28254:
----------------------------------
Because it is hard to solve this issue all in one , I file a new Jira just change `SocketChannel` to `Channel`:
https://issues.apache.org/jira/browse/SPARK-38547
> Use More Generic Netty Interfaces to Support Fabric Network in Spark
> --------------------------------------------------------------------
>
> Key: SPARK-28254
> URL: https://issues.apache.org/jira/browse/SPARK-28254
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: jiafu zhang
> Priority: Minor
>
> Spark assumes all its networks are socket though Netty has wider interface beyond socket, like Channel over SocketChannel. Socket (TCP/IP protocol) has its advantage of widely being supported. Almost all types of network, including fabric network, support socket. However, it's not as efficient as other protocols designed for fabric, like Intel OPA which has less protocol stacks and less delay than socket, not saying other more advanced features, like flow control. Thus, from view of performance, it's better to have Spark support other types of network protocols too. (we are also proposing Netty to support more fabric semantics, like RMA (Remote Memory Access) ). For fabric networks, we can use OFI (Open Fabric's Interface) 's Libfabric library to have unified API to support fabric networks from different vendors. It'll greatly reduce our effort.
> To make it possible, we need to use more generic Netty interfaces in several places in the network-common module. For example, use Channel instead of SocketChannel in TransportClientFactory and TransportContext class.
> Besides, we need to have more flexible options for Bootstrap and IOMode class. For example in createClient method in TransportClientFactory, user can pass in more options to the bootstrap instance. Currently, the options are fixed for TCP/IP.
> And in IOMode class, we can have one more option, "OFI" for initializing OFI channels based on Libfabric. Then in NettyUtils class, we can have OFI event loop group, client channel class and server channel class for "OFI" IOMode.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org