You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2020/03/16 22:36:00 UTC
[jira] [Updated] (SPARK-30850) RPC listener wildcard bindings
(IPv6)
[ https://issues.apache.org/jira/browse/SPARK-30850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-30850:
----------------------------------
Affects Version/s: (was: 2.4.5)
3.1.0
> RPC listener wildcard bindings (IPv6)
> -------------------------------------
>
> Key: SPARK-30850
> URL: https://issues.apache.org/jira/browse/SPARK-30850
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Ben Steadman
> Priority: Major
>
> Running Spark Standalone on IPv6-first infrastructure requires that RPC listeners bind IPv6 sockets e.g. {{:::7077}} instead of {{127.0.0.1:7077}} whilst advertising a public DNS name using {{SPARK_PUBLIC_DNS}}. Currently, the various RPC listeners can be configured with a *hostname/IPv4 address* (enforced via {{Utils.checkHost}}) only e.g. via {{spark.driver.bindAddress}}, {{spark.driver.host}}, {{SPARK_LOCAL_HOST}} e.t.c. which are eventually resolved using {{InetSocketAddress}} in [{{TransportServer}}|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L136]
> Initial experimentation led to a *hack* (absolutely not a real solution) to force this behaviour by setting {{bindAddress = null}} when the different `RPCEnv`s are created ([link to diff|https://gist.github.com/SteadBytes/b206335c8e2429486a13421b7cfcdc88]).
> Since {{TransportServer}} does support the wildcard variant of {{InetSocketAddress}} if {{hostToBind = null}} [here|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L137] this should be possible.
> Adding additional options similar to {{spark.driver.bindAddress}} but for other RPC components (master, driver etc) and allowing these to have wildcard/IPv6 values might be an option that avoids breaking existing configurations?
> I'm prepared to contribute the work required and upstream any changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org